Prior preferences in active inference agents: soft, hard, and goal shaping

TL;DR

This paper investigates how different preference specifications, including hard, soft, and goal shaping, affect the performance of active inference agents in navigation tasks, highlighting trade-offs between exploration and exploitation.

Contribution

It introduces four methods for defining preference distributions in active inference agents and compares their effects on learning and performance.

Findings

Goal shaping improves exploitation performance

Hard preferences hinder exploration of environment dynamics

Soft preferences balance exploration and exploitation

Abstract

Active inference proposes expected free energy as an objective for planning and decision-making to adequately balance exploitative and explorative drives in learning agents. The exploitative drive, or what an agent wants to achieve, is formalised as the Kullback-Leibler divergence between a variational probability distribution, updated at each inference step, and a preference probability distribution that indicates what states or observations are more likely for the agent, hence determining the agent's goal in a certain environment. In the literature, the questions of how the preference distribution should be specified and of how a certain specification impacts inference and learning in an active inference agent have been given hardly any attention. In this work, we consider four possible ways of defining the preference distribution, either providing the agents with hard or soft goals and either involving or not goal shaping (i.e., intermediate goals). We compare the performances of four agents, each given one of the possible preference distributions, in a grid world navigation task. Our results show that goal shaping enables the best performance overall (i.e., it promotes exploitation) while sacrificing learning about the environment's transition dynamics (i.e., it hampers exploration).