Minimax entropy: The statistical physics of optimal models

TL;DR

This paper introduces a minimax entropy principle for selecting features in models, balancing simplicity and accuracy, with applications across machine learning and biological networks, and discusses computational challenges for high-dimensional data.

Contribution

It proposes a novel minimax entropy framework for optimal feature selection based on the minimum description length principle, extending its applicability to large-scale datasets.

Findings

Identifies optimal features as those maximizing entropy with minimal entropy.

Highlights limitations of naive implementations for high-dimensional systems.

Suggests new theoretical and computational methods for large datasets.

Abstract

When constructing models of the world, we aim for optimal compressions: models that include as few details as possible while remaining as accurate as possible. But which details -- or features measured in data -- should we choose to include in a model? Here, using the minimum description length principle, we show that the optimal features are the ones that produce the maximum entropy model with minimum entropy, thus yielding a minimax entropy principle. We review applications, which range from machine learning to optimal models of biological networks. Naive implementations, however, are limited to systems with small numbers of states and features. We therefore require new theoretical insights and computational techniques to construct optimal compressions of high-dimensional datasets arising in large-scale experiments.