Foundation of Calculating Normalized Maximum Likelihood for Continuous Probability Models

TL;DR

This paper proves that the method for calculating normalized maximum likelihood (NML) code length, previously validated for discrete models, is also correct for continuous models using a novel geometric measure theory approach.

Contribution

The paper provides the first rigorous proof that the NML code length calculation method applies to continuous probability models, introducing a new decomposition technique based on the coarea formula.

Findings

Confirmed the correctness of NML code length calculation for continuous models.

Developed a novel decomposition method using the coarea formula.

Bridged the theoretical gap between discrete and continuous NML calculations.

Abstract

The normalized maximum likelihood (NML) code length is widely used as a model selection criterion based on the minimum description length principle, where the model with the shortest NML code length is selected. A common method to calculate the NML code length is to use the sum (for a discrete model) or integral (for a continuous model) of a function defined by the distribution of the maximum likelihood estimator. While this method has been proven to correctly calculate the NML code length of discrete models, no proof has been provided for continuous cases. Consequently, it has remained unclear whether the method can accurately calculate the NML code length of continuous models. In this paper, we solve this problem affirmatively, proving that the method is also correct for continuous cases. Remarkably, completing the proof for continuous cases is non-trivial in that it cannot be achieved by merely replacing the sums in discrete cases with integrals, as the decomposition trick applied to sums in the discrete model case proof is not applicable to integrals in the continuous model case proof. To overcome this, we introduce a novel decomposition approach based on the coarea formula from geometric measure theory, which is essential to establishing our proof for continuous cases.