Why Maximum Entropy? A Non-axiomatic Approach

TL;DR

This paper offers a geometric, non-axiomatic perspective on why maximum entropy methods are effective for solving ill-posed inverse problems involving probability vectors, complementing traditional axiomatic justifications.

Contribution

It introduces a geometric view of MaxEnt and ML complementarity, providing an intuitive, non-axiomatic explanation for the use of MaxEnt in inverse problems.

Findings

MaxEnt and ML are geometrically complementary.

Provides an intuitive understanding of MaxEnt without axioms.

Enhances the theoretical foundation of MaxEnt methods.

Abstract

Ill-posed inverse problems of the form y = X p where y is J-dimensional vector of a data, p is m-dimensional probability vector which cannot be measured directly and matrix X of observable variables is a known J,m matrix, J < m, are frequently solved by Shannon's entropy maximization (MaxEnt). Several axiomatizations were proposed to justify the MaxEnt method (also) in this context. The main aim of the presented work is two-fold: 1) to view the concept of complementarity of MaxEnt and Maximum Likelihood (ML) tasks from a geometric perspective, and consequently 2) to provide an intuitive and non-axiomatic answer to the 'Why MaxEnt?' question.