IHC Matters: Incorporating IHC analysis to H&E Whole Slide Image Analysis for Improved Cancer Grading via Two-stage Multimodal Bilinear Pooling Fusion

TL;DR

This paper introduces a two-stage multimodal bilinear pooling model that combines IHC and H&E images to enhance cancer grading accuracy, demonstrating significant improvements over models using only one modality.

Contribution

The study presents a novel two-stage multimodal bilinear pooling framework that effectively integrates IHC and H&E features for better cancer grading performance.

Findings

Achieves 0.953 accuracy on BCI dataset

Incorporating IHC improves grading accuracy

Multimodal fusion outperforms single-modality models

Abstract

Immunohistochemistry (IHC) plays a crucial role in pathology as it detects the over-expression of protein in tissue samples. However, there are still fewer machine learning model studies on IHC's impact on accurate cancer grading. We discovered that IHC and H\&E possess distinct advantages and disadvantages while possessing certain complementary qualities. Building on this observation, we developed a two-stage multi-modal bilinear model with a feature pooling module. This model aims to maximize the potential of both IHC and HE's feature representation, resulting in improved performance compared to their individual use. Our experiments demonstrate that incorporating IHC data into machine learning models, alongside H\&E stained images, leads to superior predictive results for cancer grading. The proposed framework achieves an impressive ACC higher of 0.953 on the public dataset BCI.