Normalized Maximum Likelihood Coding for Exponential Family with Its Applications to Optimal Clustering

TL;DR

This paper introduces a general method for calculating normalized maximum likelihood (NML) code-length for exponential families, with a focus on Gaussian mixture models, and applies it to optimal clustering, demonstrating superior performance over traditional criteria.

Contribution

It proposes a novel, efficient approach to compute NML for exponential families, especially GMMs, and applies this to determine the optimal number of clusters in data.

Findings

NML-based clustering outperforms AIC and BIC in accuracy with less data

The method efficiently computes NML for GMMs using re-normalizing techniques

Artificial data experiments validate the superiority of NML in clustering accuracy

Abstract

We are concerned with the issue of how to calculate the normalized maximum likelihood (NML) code-length. There is a problem that the normalization term of the NML code-length may diverge when it is continuous and unbounded and a straightforward computation of it is highly expensive when the data domain is finite . In previous works it has been investigated how to calculate the NML code-length for specific types of distributions. We first propose a general method for computing the NML code-length for the exponential family. Then we specifically focus on Gaussian mixture model (GMM), and propose a new efficient method for computing the NML to them. We develop it by generalizing Rissanen's re-normalizing technique. Then we apply this method to the clustering issue, in which a clustering structure is modeled using a GMM, and the main task is to estimate the optimal number of clusters on the basis of the NML code-length. We demonstrate using artificial data sets the superiority of the NML-based clustering over other criteria such as AIC, BIC in terms of the data size required for high accuracy rate to be achieved.