Automatic Breast Lesion Classification by Joint Neural Analysis of Mammography and Ultrasound

TL;DR

This paper introduces a deep learning method that combines mammography and ultrasound images to improve breast cancer lesion classification, outperforming single-modality models and matching radiologist performance.

Contribution

It presents a novel multimodal neural network approach that effectively integrates mammography and ultrasound data for enhanced breast lesion classification.

Findings

Achieved an AUC of 0.94, surpassing single-modality models.

Performed on par with an average radiologist, outperforming some radiologists.

Demonstrated consistent performance improvements with multimodal data.

Abstract

Mammography and ultrasound are extensively used by radiologists as complementary modalities to achieve better performance in breast cancer diagnosis. However, existing computer-aided diagnosis (CAD) systems for the breast are generally based on a single modality. In this work, we propose a deep-learning based method for classifying breast cancer lesions from their respective mammography and ultrasound images. We present various approaches and show a consistent improvement in performance when utilizing both modalities. The proposed approach is based on a GoogleNet architecture, fine-tuned for our data in two training steps. First, a distinct neural network is trained separately for each modality, generating high-level features. Then, the aggregated features originating from each modality are used to train a multimodal network to provide the final classification. In quantitative experiments, the proposed approach achieves an AUC of 0.94, outperforming state-of-the-art models trained over a single modality. Moreover, it performs similarly to an average radiologist, surpassing two out of four radiologists participating in a reader study. The promising results suggest that the proposed method may become a valuable decision support tool for breast radiologists.