A Fully Transformer Based Multimodal Framework for Explainable Cancer Image Segmentation Using Radiology Reports

TL;DR

Med-CTX is a transformer-based multimodal framework that integrates radiology reports and ultrasound images to improve breast cancer segmentation, interpretability, and trustworthiness in diagnosis.

Contribution

This work introduces Med-CTX, a novel fully transformer-based multimodal model that combines visual and clinical text data for explainable cancer image segmentation.

Findings

Achieves 99% Dice score and 95% IoU on BUS-BRA dataset.

Outperforms existing models like U-Net, ViT, and Swin.

Clinical reports significantly enhance segmentation accuracy and explanation quality.

Abstract

We introduce Med-CTX, a fully transformer based multimodal framework for explainable breast cancer ultrasound segmentation. We integrate clinical radiology reports to boost both performance and interpretability. Med-CTX achieves exact lesion delineation by using a dual-branch visual encoder that combines ViT and Swin transformers, as well as uncertainty aware fusion. Clinical language structured with BI-RADS semantics is encoded by BioClinicalBERT and combined with visual features utilising cross-modal attention, allowing the model to provide clinically grounded, model generated explanations. Our methodology generates segmentation masks, uncertainty maps, and diagnostic rationales all at once, increasing confidence and transparency in computer assisted diagnosis. On the BUS-BRA dataset, Med-CTX achieves a Dice score of 99% and an IoU of 95%, beating existing baselines U-Net, ViT, and Swin. Clinical text plays a key role in segmentation accuracy and explanation quality, as evidenced by ablation studies that show a -5.4% decline in Dice score and -31% in CIDEr. Med-CTX achieves good multimodal alignment (CLIP score: 85%) and increased confi dence calibration (ECE: 3.2%), setting a new bar for trustworthy, multimodal medical architecture.