Sophisticated Learning: A novel algorithm for active learning during model-based planning

TL;DR

This paper introduces Sophisticated Learning (SL), an active learning algorithm integrated into a planning framework, which improves decision-making under uncertainty by updating beliefs about model parameters during planning.

Contribution

SL is a novel algorithm that embeds active parameter learning within the Active Inference framework, enhancing planning and decision-making in uncertain environments.

Findings

SL outperforms SI and BARL in survival time and learning speed.

SL reaches convergence 40% faster than SI.

SL demonstrates robust performance in altered environments.

Abstract

We introduce Sophisticated Learning (SL), a planning-to-learn algorithm that embeds active parameter learning inside the Sophisticated Inference (SI) tree-search framework of Active Inference. Unlike SI -- which optimizes beliefs about hidden states -- SL also updates beliefs about model parameters within each simulated branch, enabling counterfactual reasoning about how future observations would improve subsequent planning. We compared SL with Bayes-adaptive Reinforcement Learning (BARL) agents as well as with its parent algorithm, SI. Using a biologically inspired seasonal foraging task in which resources shift probabilistically over a 10x10 grid, we designed experiments that forced agents to balance probabilistic reward harvesting against information gathering. In early trials, where rapid learning is vital, SL agents survive, on average, 8.2% longer than SI and 35% longer than Bayes-adaptive Reinforcement Learning. While both SL and SI showed equal convergence performance, SL reached this convergence 40% faster than SI. Additionally, SL showed robust out-performance of other algorithms in altered environment configurations. Our results show that incorporating active learning into multi-step planning materially improves decision making under radical uncertainty, and reinforces the broader utility of Active Inference for modeling biologically relevant behavior.