Learning Rate Sensitivity of Multilingual Transformer Based Large Language Models in Turkish Binary Classification

Abstract

In this study, it was aimed to analyze the factors affecting the performance of multilingual transformer based large language models in a binary classification task on a Turkish dataset. The main research question investigates how effectively multilingual models with different learning rates. The dataset used contains 5000 texts equally in the classes. Multilingual DistilBERT, multilingual BERT, RemBERT, XLM-RoBERTa Base, XLM-V, XLM-100, and ERNIE-M Base models were trained using this dataset, and their performance was evaluated using f1 score. While fine-tuning, 3 epochs, 8 batch size and AdamW optimizer settings were kept constant. In order to reveal the sensitivity of the models in the learning rate, a total of 16 different learning rates were selected in two different ranges. As a result of the trainings, it was seen that the models generally worked well at a 3e-5 learning rate. However, the RemBERT model deviated from this trend and achieved the highest performance at a 6e-6 learning rate with an f1 score of 87,90%. It was concluded that larger models were more sensitive to learning rates. © 2025 IEEE.