# Ensemble transformer-based multiple instance learning for predicting neoadjuvant chemotherapy response from breast cancer biopsy whole-slide images

## TL;DR

This paper introduces a deep learning model that uses breast cancer biopsy images to predict how well patients will respond to chemotherapy, aiming to improve treatment planning.

## Contribution

A novel Transformer-based MIL framework for NAC response prediction using biopsy images, with a two-stage classification and interpretable visualizations.

## Key findings

- The model achieved 79.3% WSI-level accuracy in internal validation and AUCs of 0.82 for pCR and 0.77 for non-response.
- External validation showed AUCs of 0.70 for pCR and 0.67 for non-response, outperforming traditional CNN models.
- Heatmap visualizations provided interpretable lesion boundaries, supporting clinical transparency and decision-making.

## Abstract

Neoadjuvant chemotherapy (NAC) is a cornerstone of breast cancer management, and accurate prediction of therapeutic efficacy is essential for optimizing treatment strategies and improving patient outcomes.

This study proposes an integrated Transformer-based Multiple Instance Learning (MIL) framework that leverages pre-treatment biopsy whole-slide images (WSIs) to predict NAC response. A multi-institutional dataset of 128 patients was collected, comprising 86 cases for training, 42 for internal validation, and 22 microscope images for external validation. The framework integrates ResNet50 feature extraction, a multi-scale attention Transformer encoder, and a two-stage classification strategy to capture both local morphological and global contextual features. Class imbalance was mitigated using SMOTE and ADASYN, while domain adaptation (DANN) and metric learning enhanced cross-modal robustness.

The proposed model achieved a WSI-level accuracy of 79.3% in internal validation and demonstrated strong discriminative ability in identifying pathological complete response (pCR, AUC = 0.82) and non-response (AUC = 0.77). External validation using lower-resolution microscope images yielded an AUC of 0.70 for pCR and 0.67 for non-response, outperforming traditional CNN architectures such as GoogleNet, ResNet34, and SqueezeNet. The model’s heatmap visualizations revealed well-defined lesion boundaries and interpretable regions of interest, underscoring its clinical transparency.

By relying solely on hematoxylin and eosin-stained WSIs, the framework provides a fully automated, interpretable, and resource-efficient approach suitable for real-world deployment. The two-stage classification design offers fine-grained stratification between pCR, partial, and poor responders, which is critical for personalized therapy planning. Future work will focus on expanding multi-center datasets and integrating advanced pathology foundation models to further enhance cross-domain generalization and clinical applicability.