Can Human Preference Redefine AI Testing? A Side-by-Side Evaluation Platform for Large Language Models

Abstract

The growing prevalence of large language models (LLMs) has made it necessary to measure the performance and output quality of these systems on an objective basis. The chief objective of this study is to develop a user-centric, interactive testing platform that evaluates LLM performance throughout various tasks. Thanks to the designed system architecture, a single input (prompt) provided by the user is simultaneously sent to multiple models in the background. The results obtained can be compared side-by-side. Data was collected from 35 volunteer users during the beta tests. It was determined that the ranking algorithm, powered by crowd-sourced votes, matches global LMSYS Chatbot Arena data at an 82.8% rate (Spearman correlation) within the scope of this pilot study. Given the initial sample size, these findings demonstrate promising local consistency rather than definitive global alignment. Additionally, the platform achieved a score of 81.93 on the System Usability Scale (SUS). Current methods generally focus on limited binary tests. This testing tool, however, combines multi-model comparison into a single interactive interface, supplying a complete evaluation perspective. © 2026 IEEE.