Jaynes & Shannon's Constrained Ignorance and Surprise

TL;DR

This paper presents a variational approach to deriving Shannon's entropy based on constraints, inspired by Jaynes's work, aiming to establish a consistent formalism where the maximum entropy principle emerges naturally.

Contribution

It introduces a novel variational framework for deriving Shannon's entropy from ignorance and constraints, expanding the theoretical understanding of entropy in physics.

Findings

Derivation of Shannon's entropy from a variational principle.

Comparison of biased and unbiased approaches to entropy.

Potential applications in theoretical and applied physics.

Abstract

In this simple article, with possible applications in theoretical and applied physics, we suggest an original way to derive the expression of Shannon's entropy from a purely variational approach,using constraints. Based on the work of Edwin T. Jaynes, our results are not fundamentally new but the context in which they are derived might, however, lead to a remarkably consistent formalism,where the maximum entropy principle appears naturally. After having given a general definition of "ignorance" in this framework, we derive the somehow general expected expression for the entropy using two approaches. In the first, one is biased and has a vague idea of the shape of the entropy function. In the second, we consider the general case, where nothing is a priori known. The merits of both ways of thinking are compared.