Notes on Generalizing the Maximum Entropy Principle to Uncertain Data

TL;DR

This paper extends the maximum entropy principle to handle uncertain data with partial observations by introducing uncertain maximum entropy and an EM-based solution, enabling broader application with black box classifiers.

Contribution

It generalizes the maximum entropy principle to uncertain data scenarios and proposes an EM-based method to solve these problems, including the use of black box classifiers.

Findings

The method effectively handles partial observations.

It generalizes existing maximum entropy principles.

The approach simplifies working with large, sparse datasets.

Abstract

The principle of maximum entropy is a broadly applicable technique for computing a distribution with the least amount of information possible constrained to match empirical data, for instance, feature expectations. We seek to generalize this principle to scenarios where the empirical feature expectations cannot be computed because the model variables are only partially observed, which introduces a dependency on the learned model. Generalizing the principle of latent maximum entropy, we introduce uncertain maximum entropy and describe an expectation-maximization based solution to approximately solve these problems. We show that our technique additionally generalizes the principle of maximum entropy. We additionally discuss the use of black box classifiers with our technique, which simplifies the process of utilizing sparse, large data sets.