Distribution-free screening of spatially variable genes in spatial transcriptomics

TL;DR

This paper introduces a novel, distribution-free method called MM-test for identifying spatially variable genes in high-dimensional spatial transcriptomics data, effectively controlling false discoveries and capturing complex 3D structures.

Contribution

The paper proposes a new SVG screening method using a quasi-likelihood ratio statistic and knockoff procedure, applicable to 2D and 3D datasets, with theoretical guarantees and superior performance.

Findings

MM-test outperforms existing methods in simulations and real datasets.

Accurately delineates fine-scale 3D brain structures.

Provides theoretical guarantees including FDR control.

Abstract

Spatial transcriptomics (ST) technologies enable transcriptome-wide gene expression profiling while preserving spatial resolution, offering unprecedented opportunities to uncover complex spatial structures. Due to the ultra-high dimensionality of ST data, identifying spatially variable genes (SVGs) associated with unknown spatial clusters has become a central task in ST data analysis. Here, we develop a distribution-free SVG screening method based on a novel quasi-likelihood ratio statistic, the MM-test, combined with a knockoff procedure to control the false discovery rate (FDR). MM-test leverages auxiliary information, such as spatial distances, about the unknown spatial domains for SVG screening. Notably, in addition to two-dimensional ST datasets, MM-test is well-suited for increasingly common three-dimensional (3D), multi-slice ST datasets. Extensive benchmarking using simulations and 34 real ST datasets demonstrates that MM-test consistently outperforms existing SVG detection methods. In a 3D mouse brain dataset, MM-test accurately delineates fine-scale structures that are challenging for other methods, such as the 3D architecture of the pyramidal layer of the hippocampal cornu ammonis and the dentate gyrus. Theoretical guarantees-including selection consistency, FDR control, and an error bound for post-selection clustering-are also established.