Finite samples inference and critical dimension for stochastically linear models

TL;DR

This paper develops finite sample bounds and Gaussian approximation results for penalized maximum likelihood estimators and posterior distributions in high-dimensional stochastic linear models, extending classical asymptotic theorems to a non-asymptotic setting.

Contribution

It introduces non-asymptotic bounds and Gaussian approximations for pMLE and posterior in large or infinite-dimensional stochastic linear models, generalizing classical asymptotic results.

Findings

Finite sample bounds for pMLE concentration and deviations

Non-asymptotic Fisher and Wilks expansions in high dimensions

Gaussian approximation of the posterior akin to Bernstein--von Mises theorem

Abstract

The aim of this note is to state a couple of general results about the properties of the penalized maximum likelihood estimators (pMLE) and of the posterior distribution for parametric models in a non-asymptotic setup and for possibly large or even infinite parameter dimension. We consider a special class of stochastically linear smooth (SLS) models satisfying two major conditions: the stochastic component of the log-likelihood is linear in the model parameter, while the expected log-likelihood is a smooth function. The main results simplify a lot if the expected log-likelihood is concave. For the pMLE, we establish a number of finite sample bounds about its concentration and large deviations as well as the Fisher and Wilks expansion. The later results extend the classical asymptotic Fisher and Wilks Theorems about the MLE to the non-asymptotic setup with large parameter dimension which can depend on the sample size. For the posterior distribution, our main result states a Gaussian approximation of the posterior which can be viewed as a finite sample analog of the prominent Bernstein--von Mises Theorem. In all bounds, the remainder is given explicitly and can be evaluated in terms of the effective sample size and effective parameter dimension. The results are dimension and coordinate free. In spite of generality, all the presented bounds are nearly sharp and the classical asymptotic results can be obtained as simple corollaries. Interesting cases of logit regression and of estimation of a log-density with smooth or truncation priors are used to specify the results and to explain the main notions.