A Thermodynamical Approach for Probability Estimation

TL;DR

This paper introduces a thermodynamics-inspired framework for probability estimation that effectively handles small sample sizes by unifying maximum likelihood and maximum entropy principles.

Contribution

It develops a novel theoretical approach based on thermodynamics concepts, addressing limitations of traditional methods in small data scenarios.

Findings

Robust probability estimation from small samples.

Unification of maximum likelihood and maximum entropy principles.

Theoretical framework based on minimum free energy.

Abstract

The issue of discrete probability estimation for samples of small size is addressed in this study. The maximum likelihood method often suffers over-fitting when insufficient data is available. Although the Bayesian approach can avoid over-fitting by using prior distributions, it still has problems with objective analysis. In response to these drawbacks, a new theoretical framework based on thermodynamics, where energy and temperature are introduced, was developed. Entropy and likelihood are placed at the center of this method. The key principle of inference for probability mass functions is the minimum free energy, which is shown to unify the two principles of maximum likelihood and maximum entropy. Our method can robustly estimate probability functions from small size data.