First-Order Representation Languages for Goal-Conditioned RL

TL;DR

This paper explores how first-order languages and goal relabeling techniques like HER can improve goal-conditioned reinforcement learning, enabling efficient learning of general policies in large, complex planning tasks with sparse rewards.

Contribution

It introduces the use of set-based goal representations and lifted subgoals to enhance HER, facilitating curriculum learning and better generalization in large-scale goal-conditioned RL.

Findings

Set-based goal representations improve learning efficiency.

Lifted subgoals enable curriculum learning with sparse rewards.

The proposed methods demonstrate computational gains on large planning problems.

Abstract

First-order relational languages have been used in MDP planning and reinforcement learning (RL) for two main purposes: specifying MDPs in compact form, and representing and learning policies that are general and not tied to specific instances or state spaces. In this work, we instead consider the use of first-order languages in goal-conditioned RL and generalized planning. The question is how to learn goal-conditioned and general policies when the training instances are large and the goal cannot be reached by random exploration alone. The technique of Hindsight Experience Replay (HER) provides an answer to this question: it relabels unsuccessful trajectories as successful ones by replacing the original goal with one that was actually achieved. If the target policy must generalize across states and goals, trajectories that do not reach the original goal states can enable more data- and time-efficient learning. In this work, we show that further performance gains can be achieved when states and goals are represented by sets of atoms. We consider three versions: goals as full states, goals as subsets of the original goals, and goals as lifted versions of these subgoals. The result is that the latter two successfully learn general policies on large planning instances with sparse rewards by automatically creating a curriculum of easier goals of increasing complexity. The experiments illustrate the computational gains of these versions, their limitations, and opportunities for addressing them.