Reinforcement Learning with Data Bootstrapping for Dynamic Subgoal Pursuit in Humanoid Robot Navigation

TL;DR

This paper introduces a hierarchical reinforcement learning framework with data bootstrapping for efficient, safe, and adaptable humanoid robot navigation in cluttered environments, combining high-level subgoal planning with low-level gait control.

Contribution

It presents a novel hierarchical approach integrating RL and MPC for dynamic subgoal pursuit, enhanced by data bootstrapping to improve training stability and efficiency.

Findings

Significantly higher navigation success rates in simulation.

Enhanced adaptability to diverse obstacle scenarios.

Faster training convergence with data bootstrapping.

Abstract

Safe and real-time navigation is fundamental for humanoid robot applications. However, existing bipedal robot navigation frameworks often struggle to balance computational efficiency with the precision required for stable locomotion. We propose a novel hierarchical framework that continuously generates dynamic subgoals to guide the robot through cluttered environments. Our method comprises a high-level reinforcement learning (RL) planner for subgoal selection in a robot-centric coordinate system and a low-level Model Predictive Control (MPC) based planner which produces robust walking gaits to reach these subgoals. To expedite and stabilize the training process, we incorporate a data bootstrapping technique that leverages a model-based navigation approach to generate a diverse, informative dataset. We validate our method in simulation using the Agility Robotics Digit humanoid across multiple scenarios with random obstacles. Results show that our framework significantly improves navigation success rates and adaptability compared to both the original model-based method and other learning-based methods.