Consistent order estimation and minimal penalties

TL;DR

This paper proves the strong consistency of a penalized likelihood method for estimating the order of nested models without requiring an upper bound, using entropy-based fluctuation analysis.

Contribution

It introduces a minimal penalty framework for order estimation that does not rely on an a priori upper bound, extending previous results.

Findings

Penalties of order η(q) log log n are sufficient for consistency.

The method applies to complex models like location mixtures.

The approach characterizes likelihood ratio fluctuations under entropy conditions.

Abstract

Consider an i.i.d. sequence of random variables whose distribution f* lies in one of a nested family of models M_q, q>=1. The smallest index q* such that M_{q*} contains f* is called the model order. We establish strong consistency of the penalized likelihood order estimator in a general setting with penalties of order \eta(q) log log n, where \eta(q) is a dimensional quantity. Moreover, such penalties are shown to be minimal. In contrast to previous work, an a priori upper bound on the model order is not assumed. The results rely on a sharp characterization of the pathwise fluctuations of the generalized likelihood ratio statistic under entropy assumptions on the model classes. Our results are applied to the geometrically complex problem of location mixture order estimation, which is widely used but poorly understood.