Goal-Reaching Policy Learning from Non-Expert Observations via Effective Subgoal Guidance

TL;DR

This paper introduces a novel subgoal guidance strategy for goal-reaching policy learning from non-expert, action-free observations, improving exploration efficiency and performance in complex robotic tasks.

Contribution

It proposes a diffusion strategy-based high-level policy for generating subgoals and integrates state-goal value functions into an off-policy framework, addressing the challenge of learning from non-expert data.

Findings

Significant performance improvements over existing methods.

Robustness to observation data corruptions.

Effective exploration in complex robotic tasks.

Abstract

In this work, we address the challenging problem of long-horizon goal-reaching policy learning from non-expert, action-free observation data. Unlike fully labeled expert data, our data is more accessible and avoids the costly process of action labeling. Additionally, compared to online learning, which often involves aimless exploration, our data provides useful guidance for more efficient exploration. To achieve our goal, we propose a novel subgoal guidance learning strategy. The motivation behind this strategy is that long-horizon goals offer limited guidance for efficient exploration and accurate state transition. We develop a diffusion strategy-based high-level policy to generate reasonable subgoals as waypoints, preferring states that more easily lead to the final goal. Additionally, we learn state-goal value functions to encourage efficient subgoal reaching. These two components naturally integrate into the off-policy actor-critic framework, enabling efficient goal attainment through informative exploration. We evaluate our method on complex robotic navigation and manipulation tasks, demonstrating a significant performance advantage over existing methods. Our ablation study further shows that our method is robust to observation data with various corruptions.