Optimal exponential bounds for aggregation of estimators for the Kullback-Leibler loss

TL;DR

This paper introduces new exponential bounds for aggregating estimators under Kullback-Leibler divergence, providing sharp oracle inequalities and demonstrating the optimality of the bounds for probabilistic models.

Contribution

It proposes novel aggregation methods based on convex combinations of logarithms of estimators, with theoretical guarantees and optimality results for density and spectral density estimation.

Findings

Sharp oracle inequalities with high probability

Optimality of the remainder bounds shown by lower bounds

Aggregation methods applicable to density and spectral density estimation

Abstract

We study the problem of model selection type aggregation with respect to the Kullback-Leibler divergence for various probabilistic models. Rather than considering a convex combination of the initial estimators $f_1, \ldots, f_N$, our aggregation procedures rely on the convex combination of the logarithms of these functions. The first method is designed for probability density estimation as it gives an aggregate estimator that is also a proper density function, whereas the second method concerns spectral density estimation and has no such mass-conserving feature. We select the aggregation weights based on a penalized maximum likelihood criterion. We give sharp oracle inequalities that hold with high probability, with a remainder term that is decomposed into a bias and a variance part. We also show the optimality of the remainder terms by providing the corresponding lower bound results.