Finite sample expansions and risk bounds in high-dimensional SLS models

TL;DR

This paper extends classical statistical results to high-dimensional SLS models, providing explicit finite sample risk bounds, distributional approximations, and conditions for valid inference in complex settings.

Contribution

It introduces finite sample expansions and risk bounds for high-dimensional SLS models, relaxing critical dimension conditions and enabling broader inference applications.

Findings

Finite sample expansions are nearly sharp and general.

Critical dimension condition relaxed from p^3 << n to p^{3/2} << n.

Results apply to classical models like logistic regression and precision matrix estimation.

Abstract

This note extends the results of classical parametric statistics like Fisher and Wilks theorem to modern setups with a high or infinite parameter dimension, limited sample size, and possible model misspecification. We consider a special class of stochastically linear smooth (SLS) models satisfying three major conditions: the stochastic component of the log-likelihood is linear in the model parameter and the expected log-likelihood is a smooth and concave function. For the penalized maximum likelihood estimators (pMLE), we establish three types of results: (1) concentration in a small vicinity of the ``truth''; (2) Fisher and Wilks expansions; (3) risk bounds. In all results, the remainder is given explicitly and can be evaluated in terms of the effective sample size and effective parameter dimension which allows us to identify the so-called \emph{critical parameter dimension}. The results are also dimension and coordinate-free. The obtained finite sample expansions are of special interest because they can be used not only for obtaining the risk bounds but also for inference, studying the asymptotic distribution, analysis of resampling procedures, etc. The main tool for all these expansions is the so-called ``basic lemma'' about linearly perturbed optimization. Despite their generality, all the presented bounds are nearly sharp and the classical asymptotic results can be obtained as simple corollaries. Our results indicate that the use of advanced fourth-order expansions allows to relax the critical dimension condition $ \mathbb{p}^{3} \ll n $ from Spokoiny (2023a) to $ \mathbb{p}^{3/2} \ll n $. Examples for classical models like logistic regression, log-density and precision matrix estimation illustrate the applicability of general results.