SIGMMA: Hierarchical Graph-Based Multi-Scale Multi-modal Contrastive Alignment of Histopathology Image and Spatial Transcriptome

TL;DR

Sigmma is a hierarchical, multi-scale contrastive learning framework that aligns histopathology images with spatial transcriptomics data across multiple scales, capturing cellular interactions and improving cross-modal predictions.

Contribution

It introduces a novel multi-scale contrastive alignment method that incorporates cell interaction graphs for better cross-modal representation learning.

Findings

Achieves 9.78% improvement in gene-expression prediction

Attains 26.93% better performance in cross-modal retrieval

Learns meaningful tissue organization in downstream tasks

Abstract

Recent advances in computational pathology have leveraged vision-language models to learn joint representations of Hematoxylin and Eosin (HE) images with spatial transcriptomic (ST) profiles. However, existing approaches typically align HE tiles with their corresponding ST profiles at a single scale, overlooking fine-grained cellular structures and their spatial organization. To address this, we propose Sigmma, a multi-modal contrastive alignment framework for learning hierarchical representations of HE images and spatial transcriptome profiles across multiple scales. Sigmma introduces multi-scale contrastive alignment, ensuring that representations learned at different scales remain coherent across modalities. Furthermore, by representing cell interactions as a graph and integrating inter- and intra-subgraph relationships, our approach effectively captures cell-cell interactions, ranging from fine to coarse, within the tissue microenvironment. We demonstrate that Sigmm learns representations that better capture cross-modal correspondences, leading to an improvement of avg. 9.78\% in the gene-expression prediction task and avg. 26.93\% in the cross-modal retrieval task across datasets. We further show that it learns meaningful multi-tissue organization in downstream analyses.