Statistical Inference for Manifold Similarity and Alignability across Noisy High-Dimensional Datasets

TL;DR

This paper introduces a robust statistical framework for comparing high-dimensional datasets supported on low-dimensional manifolds, effectively distinguishing signal from noise and enabling alignment analysis.

Contribution

It develops a spectrum-based distance measure and a statistical test for manifold similarity and alignability, with proven asymptotic properties and applicability to noisy, heterogeneous data.

Findings

The method outperforms existing approaches in robustness and power.

It accurately estimates manifold similarity in high-dimensional noisy data.

The approach is validated on multi-sample single-cell datasets.

Abstract

The rapid growth of high-dimensional datasets across various scientific domains has created a pressing need for new statistical methods to compare distributions supported on their underlying structures. Assessing similarity between datasets whose samples lie on low-dimensional manifolds requires robust techniques capable of separating meaningful signal from noise. We propose a principled framework for statistical inference of similarity and alignment between distributions supported on manifolds underlying high-dimensional datasets in the presence of heterogeneous noise. The key idea is to link the low-rank structure of observed data matrices to their underlying manifold geometry. By analyzing the spectrum of the sample covariance under a manifold signal-plus-noise model, we develop a scale-invariant distance measure between datasets based on their principal variance structures. We further introduce a consistent estimator for this distance and a statistical test for manifold alignability, and establish their asymptotic properties using random matrix theory. The proposed framework accommodates heterogeneous noise across datasets and offers an efficient, theoretically grounded approach for comparing high-dimensional datasets with low-dimensional manifold structures. Through extensive simulations and analyses of multi-sample single-cell datasets, we demonstrate that our method achieves superior robustness and statistical power compared with existing approaches.