Value of Information and Reward Specification in Active Inference and POMDPs

TL;DR

This paper analyzes how expected free energy (EFE) in active inference relates to reward-based RL, showing that EFE approximates Bayes optimal policies through information value, with implications for agent objective design.

Contribution

It provides a bottom-up analysis of EFE, demonstrating its approximation of Bayes optimal RL policies and discussing the implications for specifying objectives in active inference.

Findings

EFE approximates Bayes optimal RL policies via information value

Analysis reveals the relationship between EFE and reward-driven decision making

Implications for designing objectives in active inference agents

Abstract

Expected free energy (EFE) is a central quantity in active inference which has recently gained popularity due to its intuitive decomposition of the expected value of control into a pragmatic and an epistemic component. While numerous conjectures have been made to justify EFE as a decision making objective function, the most widely accepted is still its intuitiveness and resemblance to variational free energy in approximate Bayesian inference. In this work, we take a bottom up approach and ask: taking EFE as given, what's the resulting agent's optimality gap compared with a reward-driven reinforcement learning (RL) agent, which is well understood? By casting EFE under a particular class of belief MDP and using analysis tools from RL theory, we show that EFE approximates the Bayes optimal RL policy via information value. We discuss the implications for objective specification of active inference agents.