ViT-FuseNet: Same-Patient MRI-Pathology Feature Fusion for Multimodal Breast Cancer Diagnosis

Abstract

Background: In breast cancer diagnosis, while radiological imaging modalities provide important insights into the structural characteristics of tumors, pathological examinations remain essential for establishing a definitive diagnosis. Methods: This study proposes a Vision Transformer (ViT)-based approach developed by fusing magnetic resonance imaging (MRI) and pathology images from the same patient in breast cancer diagnosis. In the study, models combined with different classifiers were trained using ViT-16 and ViT-32 architectures. Performance of the models was evaluated using accuracy, F1 score, sensitivity, precision, ROC, AUC, and Precision-Recall (PR) curves. Results: The findings show that models multimodal image fusion models outperform single-modality models in accuracy, precision, and sensitivity, demonstrating that the fusion approach is an effective method for breast cancer diagnosis. Specifically, the ViT-B/32 (Fusion) + SVM model proved to be the most successful, achieving 92.53% accuracy and a PR-AP value of 0.9708. Conclusions: These results demonstrate that evaluating radiological and pathological images improves diagnostic accuracy and reliability, and that multimodal image fusion is effective in distinguishing malignant lesions from benign ones.