Maximum Regularized Likelihood Estimators: A General Prediction Theory and Applications

TL;DR

This paper establishes general prediction guarantees for maximum regularized likelihood estimators (MRLEs) in high-dimensional statistics, applicable to a wide range of models and regularization methods, emphasizing their broad consistency.

Contribution

It provides a unified theoretical framework for analyzing MRLEs' prediction performance using Kullback-Leibler divergence, covering convex and non-convex regularizations.

Findings

MRLEs are broadly consistent in prediction.

Results apply to tensor regression and graphical models.

Guarantees hold without restricted eigenvalue conditions.

Abstract

Maximum regularized likelihood estimators (MRLEs) are arguably the most established class of estimators in high-dimensional statistics. In this paper, we derive guarantees for MRLEs in Kullback-Leibler divergence, a general measure of prediction accuracy. We assume only that the densities have a convex parametrization and that the regularization is definite and positive homogenous. The results thus apply to a very large variety of models and estimators, such as tensor regression and graphical models with convex and non-convex regularized methods. A main conclusion is that MRLEs are broadly consistent in prediction - regardless of whether restricted eigenvalues or similar conditions hold.