Boosting MCTS with Free Energy Minimization

TL;DR

This paper introduces a novel planning framework that combines Monte Carlo Tree Search with active inference principles, enabling agents to efficiently balance exploration and exploitation by minimizing free energy in uncertain environments.

Contribution

The work integrates free energy minimization into MCTS using the Cross-Entropy Method, enhancing planning by jointly optimizing rewards and information gain.

Findings

Demonstrates improved performance over standard CEM and MCTS in continuous control tasks.

Maintains coherent estimates of value and uncertainty during planning.

Shows that the integrated approach enhances exploration efficiency.

Abstract

Active Inference, grounded in the Free Energy Principle, provides a powerful lens for understanding how agents balance exploration and goal-directed behavior in uncertain environments. Here, we propose a new planning framework, that integrates Monte Carlo Tree Search (MCTS) with active inference objectives to systematically reduce epistemic uncertainty while pursuing extrinsic rewards. Our key insight is that MCTS already renowned for its search efficiency can be naturally extended to incorporate free energy minimization by blending expected rewards with information gain. Concretely, the Cross-Entropy Method (CEM) is used to optimize action proposals at the root node, while tree expansions leverage reward modeling alongside intrinsic exploration bonuses. This synergy allows our planner to maintain coherent estimates of value and uncertainty throughout planning, without sacrificing computational tractability. Empirically, we benchmark our planner on a diverse set of continuous control tasks, where it demonstrates performance gains over both standalone CEM and MCTS with random rollouts.