Notes on the Reward Representation of Posterior Updates

TL;DR

This paper explores when decision-making updates in control and reinforcement learning can be interpreted as true Bayesian posteriors, revealing how such updates influence behavior and the ambiguity of absolute rewards.

Contribution

It provides a theoretical analysis of when KL-regularized soft updates can be exactly Bayesian posteriors within a fixed model, clarifying their informational and behavioral implications.

Findings

Posterior updates determine relative, context-dependent incentives.

Absolute rewards remain ambiguous up to baseline adjustments.

Reusing continuation values links reward descriptions across different update sequences.

Abstract

Many ideas in modern control and reinforcement learning treat decision-making as inference: start from a baseline distribution and update it when a signal arrives. We ask when this can be made literal rather than metaphorical. We study the special case where a KL-regularized soft update is exactly a Bayesian posterior inside a single fixed probabilistic model, so the update variable is a genuine channel through which information is transmitted. In this regime, behavioral change is driven only by evidence carried by that channel: the update must be explainable as an evidence reweighing of the baseline. This yields a sharp identification result: posterior updates determine the relative, context-dependent incentive signal that shifts behavior, but they do not uniquely determine absolute rewards, which remain ambiguous up to context-specific baselines. Requiring one reusable continuation value across different update directions adds a further coherence constraint linking the reward descriptions associated with different conditioning orders.