Hierarchical Active Inference using Successor Representations

TL;DR

This paper introduces a hierarchical active inference model using successor representations to improve planning and learning in complex, large-scale environments, inspired by brain's multi-scale representations.

Contribution

It presents a novel hierarchical active inference framework that combines successor representations for scalable planning and learning of abstract states and actions.

Findings

Lower-level successor representations enable learning of higher-level abstract states.

Active inference-based planning at lower levels bootstraps higher-level abstraction learning.

Higher-level abstractions facilitate more efficient planning in complex tasks.

Abstract

Active inference, a neurally-inspired model for inferring actions based on the free energy principle (FEP), has been proposed as a unifying framework for understanding perception, action, and learning in the brain. Active inference has previously been used to model ecologically important tasks such as navigation and planning, but scaling it to solve complex large-scale problems in real-world environments has remained a challenge. Inspired by the existence of multi-scale hierarchical representations in the brain, we propose a model for planning of actions based on hierarchical active inference. Our approach combines a hierarchical model of the environment with successor representations for efficient planning. We present results demonstrating (1) how lower-level successor representations can be used to learn higher-level abstract states, (2) how planning based on active inference at the lower-level can be used to bootstrap and learn higher-level abstract actions, and (3) how these learned higher-level abstract states and actions can facilitate efficient planning. We illustrate the performance of the approach on several planning and reinforcement learning (RL) problems including a variant of the well-known four rooms task, a key-based navigation task, a partially observable planning problem, the Mountain Car problem, and PointMaze, a family of navigation tasks with continuous state and action spaces. Our results represent, to our knowledge, the first application of learned hierarchical state and action abstractions to active inference in FEP-based theories of brain function.