Can Human Preference Redefine AI Testing? A Side-by-Side Evaluation Platform for Large Language Models - Aslan, Yaprak; Kaymak, Asiye; Mangan, EbrarSena; Selçuk, Ömer Faruk; Baykara, Muhammet (Institute of Electrical and Electronics Engineers Inc., 2026)
The growing prevalence of large language models (LLMs) has made it necessary to measure the performance and output quality of these systems on an objective basis. The chief objective of this study is to develop a user-centric, interactive testing platform that evaluates LLM performance throughout various tasks. Thanks to the designed system architecture, a single input (prompt) provided by the user is simultaneously sent to multiple models in the background. The results obtained can be compared side-by-side. Data was collected from 35 volunteer users during the beta tests. It was determined that the ranking algorithm, powered by crowd-sourced votes, matches global LMSYS Chatbot Arena data at an 82.8% rate (Spearman correlation) within the scope of this pilot study. Given the initial sample size, these findings demonstrate promising local consistency rather than definitive global alignment. Additionally, the platform achieved a score of 81.93 on the System Usability Scale (SUS). Current methods generally focus on limited binary tests. This testing tool, however, combines multi-model comparison into a single interactive interface, supplying a complete evaluation perspective. © 2026 IEEE.
2nd International Symposium on AI-Driven Engineering Systems, ISADES 2026 -- 19 June 2026 through 20 June 2026 -- Hybrid, Mbale -- 226193
Keywords: Ai Testing Processes; Comparative Study; Human-Preference-Based Testing; Large Language Models; Multi-Model Evaluation; Promptbased Performance Measurement; Software Quality Assurance