Active Inference for Autonomous Decision-Making with Contextual Multi-Armed Bandits

TL;DR

This paper explores using active inference as a decision-making strategy in contextual multi-armed bandits, demonstrating it can efficiently identify optimal options with fewer iterations and better regret performance.

Contribution

It introduces novel approximation methods for calculating expected free energy in active inference applied to CMABs, enabling effective decision-making under uncertainty.

Findings

Active inference requires fewer iterations to find optimal options.

It achieves superior cumulative regret compared to other strategies.

The proposed methods have low additional computational cost.

Abstract

In autonomous robotic decision-making under uncertainty, the tradeoff between exploitation and exploration of available options must be considered. If secondary information associated with options can be utilized, such decision-making problems can often be formulated as contextual multi-armed bandits (CMABs). In this study, we apply active inference, which has been actively studied in the field of neuroscience in recent years, as an alternative action selection strategy for CMABs. Unlike conventional action selection strategies, it is possible to rigorously evaluate the uncertainty of each option when calculating the expected free energy (EFE) associated with the decision agent's probabilistic model, as derived from the free-energy principle. We specifically address the case where a categorical observation likelihood function is used, such that EFE values are analytically intractable. We introduce new approximation methods for computing the EFE based on variational and Laplace approximations. Extensive simulation study results demonstrate that, compared to other strategies, active inference generally requires far fewer iterations to identify optimal options and generally achieves superior cumulative regret, for relatively low extra computational cost.