Higher-order asymptotics for the parametric complexity

TL;DR

This paper develops higher-order asymptotic expansions for the parametric complexity in exponential families, improving finite-sample accuracy over previous approximations and providing insights into information-theoretic principles in MDL.

Contribution

It derives higher-order asymptotics for parametric complexity, offering more accurate approximations and a simpler proof of Rissanen's result with an $O(n^{-1})$ rate.

Findings

Higher-order terms involve cumulants and Amari-Chentsov tensors.

The new approximations outperform Rissanen's in finite samples.

The derivation simplifies the proof of Rissanen's asymptotic result.

Abstract

The parametric complexity is the key quantity in the minimum description length (MDL) approach to statistical model selection. Rissanen and others have shown that the parametric complexity of a statistical model approaches a simple function of the Fisher information volume of the model as the sample size $n$ goes to infinity. This paper derives higher-order asymptotic expansions for the parametric complexity, in the case of exponential families and independent and identically distributed data. These higher-order approximations are calculated for some examples and are shown to have better finite-sample behaviour than Rissanen's approximation. The higher-order terms are given as expressions involving cumulants (or, more naturally, the Amari-Chentsov tensors), and these terms are likely to be interesting in themselves since they arise naturally from the general information-theoretic principles underpinning MDL. The derivation given here specializes to an alternative and arguably simpler proof of Rissanen's result (for the case considered here), proving for the first time that his approximation is $O(n^{-1})$.