Multimodal Alignment of Histopathological Images Using Cell Segmentation and Point Set Matching for Integrative Cancer Analysis

TL;DR

This paper introduces a novel framework for aligning multimodal histopathological images at the cell level using segmentation, point set matching, and graph techniques, improving integration and interpretation in cancer research.

Contribution

The proposed method combines cell segmentation with point set matching and graph refinement to achieve accurate multimodal image alignment, enabling better integration of cancer tissue data.

Findings

High alignment accuracy on ovarian cancer TMAs

Enables integration of cell-level features across modalities

Generates virtual H&E images from MxIF data

Abstract

Histopathological imaging is vital for cancer research and clinical practice, with multiplexed Immunofluorescence (MxIF) and Hematoxylin and Eosin (H&E) providing complementary insights. However, aligning different stains at the cell level remains a challenge due to modality differences. In this paper, we present a novel framework for multimodal image alignment using cell segmentation outcomes. By treating cells as point sets, we apply Coherent Point Drift (CPD) for initial alignment and refine it with Graph Matching (GM). Evaluated on ovarian cancer tissue microarrays (TMAs), our method achieves high alignment accuracy, enabling integration of cell-level features across modalities and generating virtual H&E images from MxIF data for enhanced clinical interpretation.