Hierarchical Planning Through Goal-Conditioned Offline Reinforcement Learning

TL;DR

This paper introduces a hierarchical offline reinforcement learning framework with goal-conditioned policies and model-based planning to effectively handle long-horizon, complex tasks like driving and navigation.

Contribution

It proposes a novel hierarchical planning approach combining offline RL, goal sampling, and model-based high-level planning for temporally extended tasks.

Findings

Outperforms baseline methods in long-horizon tasks

Effective handling of out-of-distribution goals

Improved planning in complex robotics scenarios

Abstract

Offline Reinforcement learning (RL) has shown potent in many safe-critical tasks in robotics where exploration is risky and expensive. However, it still struggles to acquire skills in temporally extended tasks. In this paper, we study the problem of offline RL for temporally extended tasks. We propose a hierarchical planning framework, consisting of a low-level goal-conditioned RL policy and a high-level goal planner. The low-level policy is trained via offline RL. We improve the offline training to deal with out-of-distribution goals by a perturbed goal sampling process. The high-level planner selects intermediate sub-goals by taking advantages of model-based planning methods. It plans over future sub-goal sequences based on the learned value function of the low-level policy. We adopt a Conditional Variational Autoencoder to sample meaningful high-dimensional sub-goal candidates and to solve the high-level long-term strategy optimization problem. We evaluate our proposed method in long-horizon driving and robot navigation tasks. Experiments show that our method outperforms baselines with different hierarchical designs and other regular planners without hierarchy in these complex tasks.