OncoVision: Integrating Mammography and Clinical Data through Attention-Driven Multimodal AI for Enhanced Breast Cancer Diagnosis

TL;DR

OncoVision is a multimodal AI system that combines mammography images and clinical data using attention mechanisms to improve breast cancer diagnosis, segmentation, and clinical feature prediction, with real-time reporting and global accessibility.

Contribution

It introduces a novel attention-based multimodal AI pipeline that jointly segments mammography regions and predicts clinical features, integrating imaging and clinical data for enhanced diagnosis.

Findings

Achieved state-of-the-art accuracy in segmentation tasks.

Improved diagnostic precision through late fusion strategies.

Enabled real-time, interpretable reports for clinical use.

Abstract

OncoVision is a multimodal AI pipeline that combines mammography images and clinical data for better breast cancer diagnosis. Employing an attention-based encoder-decoder backbone, it jointly segments four ROIs - masses, calcifications, axillary findings, and breast tissues - with state-of-the-art accuracy and robustly predicts ten structured clinical features: mass morphology, calcification type, ACR breast density, and BI-RADS categories. To fuse imaging and clinical insights, we developed two late-fusion strategies. By utilizing complementary multimodal data, late fusion strategies improve diagnostic precision and reduce inter-observer variability. Operationalized as a secure, user-friendly web application, OncoVision produces structured reports with dual-confidence scoring and attention-weighted visualizations for real-time diagnostic support to improve clinician trust and facilitate medical teaching. It can be easily incorporated into the clinic, making screening available in underprivileged areas around the world, such as rural South Asia. Combining accurate segmentation with clinical intuition, OncoVision raises the bar for AI-based mammography, offering a scalable and equitable solution to detect breast cancer at an earlier stage and enhancing treatment through timely interventions.