Learning Rate Sensitivity of Multilingual Transformer Based Large Language Models in Turkish Binary Classification - Alav, Selma; Das, Bihter; Benli, Kristin Surpuhi (Institute of Electrical and Electronics Engineers Inc., 2025)
In this study, it was aimed to analyze the factors affecting the performance of multilingual transformer based large language models in a binary classification task on a Turkish dataset. The main research question investigates how effectively multilingual models with different learning rates. The dataset used contains 5000 texts equally in the classes. Multilingual DistilBERT, multilingual BERT, RemBERT, XLM-RoBERTa Base, XLM-V, XLM-100, and ERNIE-M Base models were trained using this dataset, and their performance was evaluated using f1 score. While fine-tuning, 3 epochs, 8 batch size and AdamW optimizer settings were kept constant. In order to reveal the sensitivity of the models in the learning rate, a total of 16 different learning rates were selected in two different ranges. As a result of the trainings, it was seen that the models generally worked well at a 3e-5 learning rate. However, the RemBERT model deviated from this trend and achieved the highest performance at a 6e-6 learning rate with an f1 score of 87,90%. It was concluded that larger models were more sensitive to learning rates. © 2025 IEEE.
10th International Conference on Computer Science and Engineering, UBMK 2025 -- 17 September 2025 through 21 September 2025 -- Istanbul -- 214243
Keywords: learning rate sensitivity; multilingual model; natural language processing; transformer architecture; turkish classification