Comparison of human-AI agreement in ASA scoring by gender and duration of clinical experience: a real-world study

Abstract

Background Accurate preoperative risk identification is critical for patient safety and postoperative outcomes. Anaesthesiologists make decisions on the basis of ASA classification and additional parameters. Artificial intelligence (Al)-based decision support may offer more objective judgments. Methods In this retrospective multi-rater study, four anaesthesiologists and an Al system independently evaluated 1,000 cases. ASA class, postoperative ICU requirement, anaesthesia preference, intraoperative risk prediction, and additional recommendations were assessed. Concordance was analysed using Krippendorff's alpha, Cohen's kappa, Gwet's AC2, and PABAK, with percentage agreement estimated by bootstrapping. Al-physician agreement was further examined using fixed-effects logistic regression including clinician sex and professional experience as covariates. Results Physician-physician agreement was generally good to excellent across outcomes, whereas physician-Al agreement was lower and variable when assessed using K, PABAK, Gwet's ACZ, and observed agreement (P.). The highest Al concordance was observed for intraoperative risk prediction and ICU requirement, while the lowest was for anaesthesia preference. Exploratory analyses suggested that Al-physician concordance may vary by clinician experience and sex; no significant effects of sex or experience were observed for intraoperative anaesthesia-related risk prediction. Conclusion Although Al shows high concordance with physician decisions in objective/algorithmic domains, concordance remains limited in contextual and experience-based domains (anaesthesia preference). The findings support positioning Al as a safe 'second eye/warning' tool within human-in-the-loop workflows, rather than as an independent authority. Prospective, externally validated studies are needed.