Confidence Sets under Generalized Self-Concordance

TL;DR

This paper develops non-asymptotic confidence sets for estimators under generalized self-concordance, allowing for degeneracy in the Hessian and using effective dimension to improve inference in statistical models.

Contribution

It introduces a finite-sample bound for estimators based on generalized self-concordance, extending confidence set construction beyond strongly convex loss functions.

Findings

Finite-sample bounds depend on effective dimension, not parameter dimension.

Confidence sets can adapt to the optimization landscape induced by the loss.

Effective dimension can be estimated from data with quantifiable accuracy.

Abstract

This paper revisits a fundamental problem in statistical inference from a non-asymptotic theoretical viewpoint $\unicode{x2013}$ the construction of confidence sets. We establish a finite-sample bound for the estimator, characterizing its asymptotic behavior in a non-asymptotic fashion. An important feature of our bound is that its dimension dependency is captured by the effective dimension $\unicode{x2013}$ the trace of the limiting sandwich covariance $\unicode{x2013}$ which can be much smaller than the parameter dimension in some regimes. We then illustrate how the bound can be used to obtain a confidence set whose shape is adapted to the optimization landscape induced by the loss function. Unlike previous works that rely heavily on the strong convexity of the loss function, we only assume the Hessian is lower bounded at optimum and allow it to gradually becomes degenerate. This property is formalized by the notion of generalized self-concordance which originated from convex optimization. Moreover, we demonstrate how the effective dimension can be estimated from data and characterize its estimation accuracy. We apply our results to maximum likelihood estimation with generalized linear models, score matching with exponential families, and hypothesis testing with Rao's score test.