Observable Geometry of Singular Statistical Models

TL;DR

This paper introduces an invariant, geometry-based framework using observable charts to analyze singular statistical models, capturing their intrinsic structure beyond parameterization.

Contribution

It develops the concept of observable completeness and observable order to characterize model distinguishability intrinsically, extending classical asymptotic theory to singular models.

Findings

Observable order bounds the rate of Kullback-Leibler divergence decay.

Observable coordinates reveal identifiable and degenerate structures in models.

Framework applies to reduced-rank regression and Gaussian mixture models.

Abstract

Singular statistical models arise whenever different parameter values induce the same distribution, leading to non-identifiability and a breakdown of classical asymptotic theory. While existing approaches analyze these phenomena in parameter space, the resulting descriptions depend heavily on parameterization and obscure the intrinsic statistical structure of the model. In this paper, we introduce an invariant framework based on \emph{observable charts}: collections of functionals of the data distribution that distinguish probability measures. These charts define local coordinate systems directly on the model space, independent of parameterization. We formalize \emph{observable completeness} as the ability of such charts to detect identifiable directions, and introduce \emph{observable order} to quantify higher-order distinguishability along analytic perturbations. Our main result establishes that, under mild regularity conditions, observable order provides a lower bound on the rate at which Kullback-Leibler divergence vanishes along analytic paths. This connects intrinsic geometric structure in model space to statistical distinguishability and recovers classical behavior in regular models while extending naturally to singular settings. We illustrate the framework in reduced-rank regression and Gaussian mixture models, where observable coordinates reveal both identifiable structure and singular degeneracies. These results suggest that observable charts provide a unified and parameterization-invariant language for studying singular models and offer a pathway toward intrinsic formulations of invariants such as learning coefficients.