Multimodal Pathology Image Search Between H&E Slides and Multiplexed Immunofluorescent Images

TL;DR

This paper introduces a novel multimodal pathology image search method that aligns H&E slides with multiplexed immunofluorescent images using VAE and DTW, enhancing diagnostic accuracy and personalized therapy.

Contribution

The approach uniquely combines VAE and DTW for multimodal image alignment, improving differential diagnosis and therapeutic decision-making in pathology.

Findings

Method successfully aligns H&E and mIF images in colorectal cancer and tonsil samples.

System demonstrates robustness despite class imbalance in data.

Ablation study validates the correlation mapping between modalities.

Abstract

We present an approach for multimodal pathology image search, using dynamic time warping (DTW) on Variational Autoencoder (VAE) latent space that is fed into a ranked choice voting scheme to retrieve multiplexed immunofluorescent imaging (mIF) that is most similar to a query H&E slide. Through training the VAE and applying DTW, we align and compare mIF and H&E slides. Our method improves differential diagnosis and therapeutic decisions by integrating morphological H&E data with immunophenotyping from mIF, providing clinicians a rich perspective of disease states. This facilitates an understanding of the spatial relationships in tissue samples and could revolutionize the diagnostic process, enhancing precision and enabling personalized therapy selection. Our technique demonstrates feasibility using colorectal cancer and healthy tonsil samples. An exhaustive ablation study was conducted on a search engine designed to explore the correlation between multiplexed Immunofluorescence (mIF) and Hematoxylin and Eosin (H&E) staining, in order to validate its ability to map these distinct modalities into a unified vector space. Despite extreme class imbalance, the system demonstrated robustness and utility by returning similar results across various data features, which suggests potential for future use in multimodal histopathology data analysis.