Multiclass Turkish Sentiment Classification on Turkish E-Commerce Reviews Using Transformer Models

Abstract

This study presents a comparative analysis of transformer-based models for multi-class sentiment analysis in Turkish natural language processing. As the data source, a subset of 15,000 randomly selected samples was used from the TRSAv1 dataset, which contains 150,000 user reviews labeled as positive, negative, or neutral, collected from Turkish e-commerce platforms. Five different transformer models - BERTurk, DeBERTa, RoBERTa, XLM-R, and ELECTRA - were trained on this dataset. Each model was trained for 20 epochs, and performance evaluations were conducted using metrics such as accuracy, class-based F1 scores, and confusion matrices on a held-out test set. Among the models, XLM-R 83.1% accuracy, 0.82 macro F1-score and RoBERTa 82.0% accuracy, 0.81 macro F1-score produced more balanced results across sentiment classes and performed better in accurately classifying neutral reviews. RoBERTa, in particular, achieved 752 correct classifications in the positive class, with only 52 misclassifications as negative. XLM-R yielded the best performance in the neutral class, correctly identifying 661 instances. BERTurk achieved high performance in the classification of negative reviews, but performed relatively lower in distinguishing neutral samples 78.2% accuracy, 0.76 macro F1-score. DeBERTa 72.7% accuracy, 0.71 macro F1 and ELECTRA 73.6% accuracy, 0.70 macro F1 had the lowest overall accuracy among the models and struggled especially with neutral class predictions. © 2025 IEEE.