Subgoal Discovery Using a Free Energy Paradigm and State Aggregations

TL;DR

This paper introduces a novel subgoal discovery method in reinforcement learning using a free energy paradigm, which identifies unpredictable states for hierarchical task decomposition without prior knowledge, applicable even in stochastic environments.

Contribution

It proposes a new approach leveraging free energy to discover subgoals based on state unpredictability, enhancing hierarchical RL methods.

Findings

Effective subgoal discovery in navigation tasks

Robust performance in stochastic environments

No prior task knowledge required

Abstract

Reinforcement learning (RL) plays a major role in solving complex sequential decision-making tasks. Hierarchical and goal-conditioned RL are promising methods for dealing with two major problems in RL, namely sample inefficiency and difficulties in reward shaping. These methods tackle the mentioned problems by decomposing a task into simpler subtasks and temporally abstracting a task in the action space. One of the key components for task decomposition of these methods is subgoal discovery. We can use the subgoal states to define hierarchies of actions and also use them in decomposing complex tasks. Under the assumption that subgoal states are more unpredictable, we propose a free energy paradigm to discover them. This is achieved by using free energy to select between two spaces, the main space and an aggregation space. The $model \; changes$ from neighboring states to a given state shows the unpredictability of a given state, and therefore it is used in this paper for subgoal discovery. Our empirical results on navigation tasks like grid-world environments show that our proposed method can be applied for subgoal discovery without prior knowledge of the task. Our proposed method is also robust to the stochasticity of environments.