Multi stain graph fusion for multimodal integration in pathology

TL;DR

This paper presents a multimodal graph fusion method combining CNN and GNN to integrate multiple histological stains for improved pathology scoring, demonstrating significant accuracy gains in NASH assessment.

Contribution

It introduces a novel CNN-GNN based graph fusion approach that leverages complementary information from multiple non-registered pathology images for better disease scoring.

Findings

Up to 20% improvement in fibrosis and NAS prediction accuracy.

Effective integration of multi-stain histology images using graph fusion.

Demonstrates the value of multimodal data in histologic assessment.

Abstract

In pathology, tissue samples are assessed using multiple staining techniques to enhance contrast in unique histologic features. In this paper, we introduce a multimodal CNN-GNN based graph fusion approach that leverages complementary information from multiple non-registered histopathology images to predict pathologic scores. We demonstrate this approach in nonalcoholic steatohepatitis (NASH) by predicting CRN fibrosis stage and NAFLD Activity Score (NAS). Primary assessment of NASH typically requires liver biopsy evaluation on two histological stains: Trichrome (TC) and hematoxylin and eosin (H&E). Our multimodal approach learns to extract complementary information from TC and H&E graphs corresponding to each stain while simultaneously learning an optimal policy to combine this information. We report up to 20% improvement in predicting fibrosis stage and NAS component grades over single-stain modeling approaches, measured by computing linearly weighted Cohen's kappa between machine-derived vs. pathologist consensus scores. Broadly, this paper demonstrates the value of leveraging diverse pathology images for improved ML-powered histologic assessment.