Spatially Gene Expression Prediction using Dual-Scale Contrastive Learning

TL;DR

This paper introduces NH2ST, a dual-scale contrastive learning framework that integrates spatial context and multimodal data to improve gene expression prediction from pathology images, outperforming existing methods.

Contribution

The novel NH2ST framework effectively captures spatial and cross-modal relationships using contrastive learning, advancing gene expression prediction accuracy from pathology images.

Findings

Achieves over 20% improvement in PCC metrics on six datasets.

Effectively models spatial and multimodal interactions.

Outperforms existing gene expression prediction methods.

Abstract

Spatial transcriptomics (ST) provides crucial insights into tissue micro-environments, but is limited to its high cost and complexity. As an alternative, predicting gene expression from pathology whole slide images (WSI) is gaining increasing attention. However, existing methods typically rely on single patches or a single pathology modality, neglecting the complex spatial and molecular interactions between target and neighboring information (e.g., gene co-expression). This leads to a failure in establishing connections among adjacent regions and capturing intricate cross-modal relationships. To address these issues, we propose NH2ST, a framework that integrates spatial context and both pathology and gene modalities for gene expression prediction. Our model comprises a query branch and a neighbor branch to process paired target patch and gene data and their neighboring regions, where cross-attention and contrastive learning are employed to capture intrinsic associations and ensure alignments between pathology and gene expression. Extensive experiments on six datasets demonstrate that our model consistently outperforms existing methods, achieving over 20% in PCC metrics. Codes are available at https://github.com/MCPathology/NH2ST