Higher-Order Equilibrium Tracking for EM-Compressible Online Estimation

TL;DR

This paper develops a theoretical framework for online estimation in latent-variable models, focusing on tracking empirical equilibrium points and providing convergence guarantees with finite-sample risk bounds.

Contribution

It introduces a novel decomposition of online estimates into equilibrium and tracking lag, along with a batch-to-online transfer theorem and equilibrium-jet predictors for improved convergence.

Findings

Proves that online estimators inherit batch CLT under certain conditions.

Develops equilibrium-jet predictors with localized tracking rates.

Demonstrates the theory in latent Gaussian covariance estimation with risk envelopes.

Abstract

We study online estimation in latent-variable models by recasting the problem as tracking a moving empirical equilibrium. Standard online EM and stochastic approximation analyses primarily study convergence toward the population parameter and typically do not isolate the empirical batch optimum from the online tracking error at finite horizon. Our framework decomposes the online estimate into the frozen batch equilibrium at the current running statistic and a tracking lag that captures the algorithm's delay behind this moving target. We prove a batch-to-online transfer theorem: provided $\lVert e_T \rVert_{L^{2}} = o(T^{-1/2})$, the online estimator inherits the batch central limit theorem and the sharp first-order risk constant. Our key observation is that the empirical optimum evolves on a smooth equilibrium manifold indexed by the running statistic. An $m$-th order equilibrium-jet predictor combined with an order-$\nu$ frozen corrector yields localized tracking rates $O(T^{-\nu(m+1)})$. We formalize EM-compressibility and EM-jet$^R$-compressibility as the structural conditions that make the equilibrium response and the Newton corrector evaluable from a retained streaming statistic. The theory is instantiated in latent linear Gaussian covariance estimation, where the first-order scheme operates on a compressed $d \times d$ statistic with explicit finite-sample risk envelopes and a certified restart rule.