Minimum Description Length Principle for Maximum Entropy Model Selection

TL;DR

This paper introduces an MDL-based approach for selecting maximum entropy models, deriving NML codelengths, and demonstrating its application to gene selection with promising simulation results.

Contribution

It formulates maximum entropy model selection as an MDL problem and derives the NML codelength for these models, connecting it to the minimax entropy principle.

Findings

Derived NML codelengths for maximum entropy models

Proved minimax entropy as a special case of model selection

Applied method successfully to gene selection problem

Abstract

Model selection is central to statistics, and many learning problems can be formulated as model selection problems. In this paper, we treat the problem of selecting a maximum entropy model given various feature subsets and their moments, as a model selection problem, and present a minimum description length (MDL) formulation to solve this problem. For this, we derive normalized maximum likelihood (NML) codelength for these models. Furthermore, we prove that the minimax entropy principle is a special case of maximum entropy model selection, where one assumes that complexity of all the models are equal. We apply our approach to gene selection problem and present simulation results.