On Maximum Entropy Density Estimation with Relaxed Moment Constraints

TL;DR

This paper provides a mathematical framework for estimating probability distributions using maximum entropy principles with relaxed constraints on continuous domains.

Contribution

The paper introduces a rigorous convex-analytic treatment of relaxed maximum entropy problems in continuous domains without discretization.

Findings

The infinite-dimensional primal problem can be reduced to a dual formulation with finitely many variables.

Maximum Entropy solutions are characterized in exponential form under mild qualification conditions.

The framework unifies exact and relaxed moment constraints in a single variational formulation.

Abstract

We study Maximum Entropy density estimation on continuous domains under finitely many moment constraints, formulated as the minimization of the Kullback–Leibler divergence with respect to a reference measure. To model uncertainty in empirical moments, constraints are relaxed through convex penalty functions, leading to an infinite-dimensional convex optimization problem over probability densities. The main contribution of this work is a rigorous convex-analytic treatment of such relaxed Maximum Entropy problems in a functional setting, without discretization or smoothness assumptions on the density. Using convex integral functionals and an extension of Fenchel duality, we show that, under mild and explicit qualification conditions, the infinite-dimensional primal problem admits a dual formulation involving only finitely many variables. This reduction can be interpreted as a continuous-domain instance of partially finite convex programming. The resulting dual problem yields explicit primal–dual optimality conditions and characterizes Maximum Entropy solutions in exponential form. The proposed framework unifies exact and relaxed moment constraints, including box and quadratic relaxations, within a single variational formulation, and provides a mathematically sound foundation for relaxed Maximum Entropy methods previously studied mainly in finite or discrete settings. A brief numerical illustration demonstrates the practical tractability of the approach.

Figures