Active Inference for Stochastic Control

TL;DR

This paper evaluates active inference as a control method in stochastic environments, demonstrating its advantages over reinforcement learning through simulations of complex, partially observable tasks.

Contribution

It extends active inference to stochastic control settings by incorporating recent planning algorithms and demonstrates its effectiveness in complex, partially observable environments.

Findings

Active inference outperforms reinforcement learning in stochastic control tasks.

The approach effectively handles environment stochasticity and partial observability.

Simulation results validate the utility of active inference in complex control scenarios.

Abstract

Active inference has emerged as an alternative approach to control problems given its intuitive (probabilistic) formalism. However, despite its theoretical utility, computational implementations have largely been restricted to low-dimensional, deterministic settings. This paper highlights that this is a consequence of the inability to adequately model stochastic transition dynamics, particularly when an extensive policy (i.e., action trajectory) space must be evaluated during planning. Fortunately, recent advancements propose a modified planning algorithm for finite temporal horizons. We build upon this work to assess the utility of active inference for a stochastic control setting. For this, we simulate the classic windy grid-world task with additional complexities, namely: 1) environment stochasticity; 2) learning of transition dynamics; and 3) partial observability. Our results demonstrate the advantage of using active inference, compared to reinforcement learning, in both deterministic and stochastic settings.