mTREE: Multi-Level Text-Guided Representation End-to-End Learning for Whole Slide Image Analysis

TL;DR

mTREE is an innovative end-to-end framework that integrates multi-scale histopathology image features with textual data, enhancing slide analysis and prediction tasks.

Contribution

This paper introduces mTREE, a novel multi-level text-guided learning approach that seamlessly combines image and text data for whole slide image analysis.

Findings

Outperforms baseline methods in classification tasks.

Achieves superior survival prediction accuracy.

Effectively localizes key histopathology regions using textual guidance.

Abstract

Multi-modal learning adeptly integrates visual and textual data, but its application to histopathology image and text analysis remains challenging, particularly with large, high-resolution images like gigapixel Whole Slide Images (WSIs). Current methods typically rely on manual region labeling or multi-stage learning to assemble local representations (e.g., patch-level) into global features (e.g., slide-level). However, there is no effective way to integrate multi-scale image representations with text data in a seamless end-to-end process. In this study, we introduce Multi-Level Text-Guided Representation End-to-End Learning (mTREE). This novel text-guided approach effectively captures multi-scale WSI representations by utilizing information from accompanying textual pathology information. mTREE innovatively combines - the localization of key areas (global-to-local) and the development of a WSI-level image-text representation (local-to-global) - into a unified, end-to-end learning framework. In this model, textual information serves a dual purpose: firstly, functioning as an attention map to accurately identify key areas, and secondly, acting as a conduit for integrating textual features into the comprehensive representation of the image. Our study demonstrates the effectiveness of mTREE through quantitative analyses in two image-related tasks: classification and survival prediction, showcasing its remarkable superiority over baselines.