Likelihood-Preserving Embeddings for Statistical Inference

TL;DR

This paper introduces a theoretical framework for likelihood-preserving embeddings that maintain the integrity of likelihood-based inference when data is compressed into learned representations, with practical neural network methods and validation on distributions and clinical data.

Contribution

It develops the Likelihood-Ratio Distortion metric and the Hinge Theorem, providing conditions under which embeddings preserve statistical inference, and offers a neural network-based constructive approach.

Findings

Controlling the distortion $ ext{Δ}_n$ preserves likelihood-based tests.

Neural network embeddings can approximate sufficient statistics with provable guarantees.

Experiments confirm the phase transition and practical effectiveness of the method.

Abstract

Modern machine learning embeddings provide powerful compression of high-dimensional data, yet they typically destroy the geometric structure required for classical likelihood-based statistical inference. This paper develops a rigorous theory of likelihood-preserving embeddings: learned representations that can replace raw data in likelihood-based workflows -- hypothesis testing, confidence interval construction, model selection -- without altering inferential conclusions. We introduce the Likelihood-Ratio Distortion metric $\Delta_n$, which measures the maximum error in log-likelihood ratios induced by an embedding. Our main theoretical contribution is the Hinge Theorem, which establishes that controlling $\Delta_n$ is necessary and sufficient for preserving inference. Specifically, if the distortion satisfies $\Delta_n = o_p(1)$, then (i) all likelihood-ratio based tests and Bayes factors are asymptotically preserved, and (ii) surrogate maximum likelihood estimators are asymptotically equivalent to full-data MLEs. We prove an impossibility result showing that universal likelihood preservation requires essentially invertible embeddings, motivating the need for model-class-specific guarantees. We then provide a constructive framework using neural networks as approximate sufficient statistics, deriving explicit bounds connecting training loss to inferential guarantees. Experiments on Gaussian and Cauchy distributions validate the sharp phase transition predicted by exponential family theory, and applications to distributed clinical inference demonstrate practical utility.