Goal-Conditioned Reinforcement Learning with Disentanglement-based Reachability Planning

TL;DR

This paper introduces REPlan, a goal-conditioned reinforcement learning method with disentangled representations and reachability discrimination, significantly improving performance in temporally extended tasks with high-dimensional observations.

Contribution

The paper proposes a novel REPlan algorithm combining a disentangled representation module and a reachability discrimination module for efficient goal-reaching in high-dimensional spaces.

Findings

REPlan outperforms state-of-the-art methods in vision-based simulation tasks.

REPlan effectively learns compact representations for high-dimensional observations.

REPlan demonstrates successful real-world task performance.

Abstract

Goal-Conditioned Reinforcement Learning (GCRL) can enable agents to spontaneously set diverse goals to learn a set of skills. Despite the excellent works proposed in various fields, reaching distant goals in temporally extended tasks remains a challenge for GCRL. Current works tackled this problem by leveraging planning algorithms to plan intermediate subgoals to augment GCRL. Their methods need two crucial requirements: (i) a state representation space to search valid subgoals, and (ii) a distance function to measure the reachability of subgoals. However, they struggle to scale to high-dimensional state space due to their non-compact representations. Moreover, they cannot collect high-quality training data through standard GC policies, which results in an inaccurate distance function. Both affect the efficiency and performance of planning and policy learning. In the paper, we propose a goal-conditioned RL algorithm combined with Disentanglement-based Reachability Planning (REPlan) to solve temporally extended tasks. In REPlan, a Disentangled Representation Module (DRM) is proposed to learn compact representations which disentangle robot poses and object positions from high-dimensional observations in a self-supervised manner. A simple REachability discrimination Module (REM) is also designed to determine the temporal distance of subgoals. Moreover, REM computes intrinsic bonuses to encourage the collection of novel states for training. We evaluate our REPlan in three vision-based simulation tasks and one real-world task. The experiments demonstrate that our REPlan significantly outperforms the prior state-of-the-art methods in solving temporally extended tasks.