HER2 Expression Prediction with Flexible Multi-Modal Inputs via Dynamic Bidirectional Reconstruction

TL;DR

This paper introduces a flexible multi-modal framework for HER2 prediction in breast cancer that adapts to available imaging modalities, significantly improving accuracy and accessibility in resource-limited settings.

Contribution

It presents a novel dynamic branch selector and cross-modal GAN enabling accurate HER2 prediction with either single or dual imaging modalities, reducing reliance on costly dual-modality data.

Findings

H&E-only accuracy improved to 94.25%

Full dual-modality accuracy achieved 95.09%

Single-modality reliability maintained at 90.28%

Abstract

In breast cancer HER2 assessment, clinical evaluation relies on combined H&E and IHC images, yet acquiring both modalities is often hindered by clinical constraints and cost. We propose an adaptive bimodal prediction framework that flexibly supports single- or dual-modality inputs through two core innovations: a dynamic branch selector activating modality completion or joint inference based on input availability, and a cross-modal GAN (CM-GAN) enabling feature-space reconstruction of missing modalities. This design dramatically improves H&E-only accuracy from 71.44% to 94.25%, achieves 95.09% with full dual-modality inputs, and maintains 90.28% reliability under single-modality conditions. The "dual-modality preferred, single-modality compatible" architecture delivers near-dual-modality accuracy without mandatory synchronized acquisition, offering a cost-effective solution for resource-limited regions and significantly improving HER2 assessment accessibility.