SHAP-CAT: A interpretable multi-modal framework enhancing WSI classification via virtual staining and shapley-value-based multimodal fusion

TL;DR

SHAP-CAT is an interpretable multimodal framework that improves whole slide image classification by using virtual staining and Shapley-value-based fusion to select important features, leading to significant accuracy gains.

Contribution

The paper introduces SHAP-CAT, a novel interpretable multimodal framework that employs Shapley-value-based dimension reduction and virtual staining for enhanced histopathology classification.

Findings

Achieved up to 11% accuracy improvement on histopathology datasets.

Demonstrated effective feature importance attribution via Shapley values.

Enhanced model interpretability with multimodal fusion and synthetic data.

Abstract

The multimodal model has demonstrated promise in histopathology. However, most multimodal models are based on H\&E and genomics, adopting increasingly complex yet black-box designs. In our paper, we propose a novel interpretable multimodal framework named SHAP-CAT, which uses a Shapley-value-based dimension reduction technique for effective multimodal fusion. Starting with two paired modalities -- H\&E and IHC images, we employ virtual staining techniques to enhance limited input data by generating a new clinical-related modality. Lightweight bag-level representations are extracted from image modalities and a Shapley-value-based mechanism is used for dimension reduction. For each dimension of the bag-level representation, attribution values are calculated to indicate how changes in the specific dimensions of the input affect the model output. In this way, we select a few top important dimensions of bag-level representation for each image modality to late fusion. Our experimental results demonstrate that the proposed SHAP-CAT framework incorporating synthetic modalities significantly enhances model performance, yielding a 5\% increase in accuracy for the BCI, an 8\% increase for IHC4BC-ER, and an 11\% increase for the IHC4BC-PR dataset.