Asymptotic Log-loss of Prequential Maximum Likelihood Codes

TL;DR

This paper investigates the asymptotic redundancy of prequential maximum likelihood codes within exponential family models, revealing that their growth rate differs from other universal codes and may be problematic for MDL model selection.

Contribution

It provides a precise characterization of the redundancy growth rate for prequential ML codes and compares it to other universal coding schemes, highlighting potential issues for model selection.

Findings

Redundancy grows at rate (v1/v2)/2 ln n, where v1 and v2 are variances.

Prequential codes differ from MDL, Shtarkov, and Bayes codes in their asymptotic behavior.

This difference can be undesirable for MDL-based model selection.

Abstract

We analyze the Dawid-Rissanen prequential maximum likelihood codes relative to one-parameter exponential family models M. If data are i.i.d. according to an (essentially) arbitrary P, then the redundancy grows at rate c/2 ln n. We show that c=v1/v2, where v1 is the variance of P, and v2 is the variance of the distribution m* in M that is closest to P in KL divergence. This shows that prequential codes behave quite differently from other important universal codes such as the 2-part MDL, Shtarkov and Bayes codes, for which c=1. This behavior is undesirable in an MDL model selection setting.