Finite-sample risk bounds for maximum likelihood estimation with arbitrary penalties

TL;DR

This paper extends finite-sample risk bounds to unpenalized and small-penalty maximum likelihood estimators, providing exact $1/n$ bounds for iid parametric models and discussing implications for adaptive estimation.

Contribution

It introduces a general inequality applicable to arbitrary penalties, enabling precise risk bounds for MLE without penalties and improving existing bounds.

Findings

Exact $1/n$ risk bounds for iid parametric models.

Extension of bounds to unpenalized MLE and small penalties.

Implications for adaptive estimation methods.

Abstract

The MDL two-part coding $ \textit{index of resolvability} $ provides a finite-sample upper bound on the statistical risk of penalized likelihood estimators over countable models. However, the bound does not apply to unpenalized maximum likelihood estimation or procedures with exceedingly small penalties. In this paper, we point out a more general inequality that holds for arbitrary penalties. In addition, this approach makes it possible to derive exact risk bounds of order $1/n$ for iid parametric models, which improves on the order $(\log n)/n$ resolvability bounds. We conclude by discussing implications for adaptive estimation.