Optimal information deletion and Bayes' theorem

TL;DR

This paper revisits Bayes' theorem from the perspective of optimal information deletion, proving that the leave-data-out posterior aligns with the optimal deletion rule.

Contribution

It establishes that the optimal information deletion rule is equivalent to the leave-data-out posterior derived from Bayes' theorem.

Findings

The optimal information deletion rule coincides with the leave-data-out posterior.

Revisits foundational ideas of Bayes' theorem from an information deletion perspective.

Connects variational inference with data removal processes.

Abstract

Arnold Zellner published a seminal paper on Bayes' theorem as an optimal information processing rule, a result that led to the variational formulation of Bayes' theorem, and a central idea in generalized variational inference. Almost 40 years later, we revisit these ideas, but from the perspective of information deletion. We investigate rules that update a posterior distribution into an antedata distribution when a portion of data is removed. In such context, a rule that does not destroy or create nonexistent information is called the optimal information deletion rule and we prove that it coincides with the leave-data-out posterior from Bayes' theorem.