An End-to-End Framework for Surgical Phase Recognition and Large Language Model-Based Decision Support in Medical Videos

Abstract

Automatic recognition of surgical workflow phases from laparoscopic videos is a key enabler for context-aware intraoperative assistance, surgical training, and safety monitoring. While recent deep learning approaches have achieved promising accuracy on benchmark datasets, most systems remain limited to isolated phase classification without providing actionable guidance or interpretability. In this paper, we propose an end-to-end framework that integrates EfficientNet-B0-based surgical phase recognition with large language model (LLM)-driven surgical next-step guidance. The system operates on individual laparoscopic frames and produces both phase predictions and structured, phase-aware recommendations through configurable behavioral profiles. Experiments conducted on the Cholec80 dataset demonstrate that EfficientNet-B0 achieves strong validation performance, with a validation accuracy of 85.87% and a weighted F1-score of 85.95%. Phase-wise analysis and confusion matrix evaluation reveal predictable ambiguities in early and transitional phases, while Grad-CAM visualizations provide interpretable evidence that the model attends to anatomically and procedurally relevant regions. A lightweight web-based prototype illustrates how visual predictions can be transformed into context-aware textual guidance. By unifying perception, explainability, and language-based reasoning within a single deployable pipeline, this work advances toward practical and interpretable surgical assistance systems for laparoscopic procedures. © 2026 IEEE.