SENCA-st: Integrating Spatial Transcriptomics and Histopathology with Cross Attention Shared Encoder for Region Identification in Cancer Pathology

TL;DR

This paper introduces SENCA-st, a novel model that effectively integrates spatial transcriptomics and histopathology data using cross-attention, improving tumor region identification and heterogeneity analysis in cancer pathology.

Contribution

The paper presents a new shared encoder architecture with cross-attention that preserves and emphasizes both structural and functional features in multimodal cancer data.

Findings

Outperforms existing methods in tumor region detection

Accurately identifies tumor heterogeneity and micro-environment regions

Enhances understanding of cancer tissue complexity

Abstract

Spatial transcriptomics is an emerging field that enables the identification of functional regions based on the spatial distribution of gene expression. Integrating this functional information present in transcriptomic data with structural data from histopathology images is an active research area with applications in identifying tumor substructures associated with cancer drug resistance. Current histopathology-spatial-transcriptomic region segmentation methods suffer due to either making spatial transcriptomics prominent by using histopathology features just to assist processing spatial transcriptomics data or using vanilla contrastive learning that make histopathology images prominent due to only promoting common features losing functional information. In both extremes, the model gets either lost in the noise of spatial transcriptomics or overly smoothed, losing essential information. Thus, we propose our novel architecture SENCA-st (Shared Encoder with Neighborhood Cross Attention) that preserves the features of both modalities. More importantly, it emphasizes regions that are structurally similar in histopathology but functionally different on spatial transcriptomics using cross-attention. We demonstrate the superior performance of our model that surpasses state-of-the-art methods in detecting tumor heterogeneity and tumor micro-environment regions, a clinically crucial aspect.