Spatially Regularized Gaussian Mixtures for Clustering Spatial Transcriptomic Data

TL;DR

This paper introduces a novel clustering method for spatial transcriptomic data using Gaussian processes and a generalized Cholesky decomposition to better capture non-stationary spatial correlations.

Contribution

It presents a flexible modeling framework that improves gene clustering by incorporating spatial structure without kernel misspecification issues.

Findings

Identifies gene clusters with distinct spatial correlation patterns.

Handles non-stationary spatial covariance structures effectively.

Applied successfully to tissue data revealing biologically relevant clusters.

Abstract

Spatial transcriptomics measures the expression of thousands of genes in a tissue sample while preserving its spatial structure. This class of technologies has enabled the investigation of the spatial variation of gene expressions and their impact on specific biological processes. Identifying genes with similar expression profiles is of utmost importance, thus motivating the development of flexible methods leveraging spatial data structure to cluster genes. Here, we propose a modeling framework for clustering observations measured over numerous spatial locations via Gaussian processes. Rather than specifying their covariance kernels as a function of the spatial structure, we use it to inform a generalized Cholesky decomposition of their precision matrices. This approach prevents issues with kernel misspecification and facilitates the estimation of a non-stationarity spatial covariance structure. Applied to spatial transcriptomic data, our model identifies gene clusters with distinctive spatial correlation patterns across tissue areas comprising different cell types, like tumoral and stromal areas.