Finite sample improvement of Akaike's Information Criterion

TL;DR

This paper introduces an improved version of Akaike's Information Criterion that reduces overfitting in small samples by incorporating an over-penalization approach, supported by theoretical guarantees and empirical results.

Contribution

It proposes a novel over-penalization modification to AIC that enhances model selection accuracy, especially in small sample scenarios, with proven nonasymptotic optimality.

Findings

The modified criterion avoids overfitting in small samples.

It achieves sharp oracle inequalities in density estimation.

Experimental results outperform AICc in bin size selection.

Abstract

We emphasize that it is possible to improve the principle of unbiased risk estimation for model selection by addressing excess risk deviations in the design of penalization procedures. Indeed, we propose a modification of Akaike's Information Criterion that avoids overfitting, even when the sample size is small. We call this correction an over-penalization procedure. As proof of concept, we show the nonasymptotic optimality of our histogram selection procedure in density estimation by establishing sharp oracle inequalities for the Kullback-Leibler divergence. One of the main features of our theoretical results is that they include the estimation of unbounded logdensities. To do so, we prove several analytical and probabilistic lemmas that are of independent interest. In an experimental study, we also demonstrate state-of-the-art performance of our over-penalization criterion for bin size selection, in particular outperforming AICc procedure.