TDMPBC: Self-Imitative Reinforcement Learning for Humanoid Robot Control

TL;DR

This paper introduces SIRL, a reinforcement learning framework for humanoid robots that imitates task-relevant trajectories to improve learning efficiency and success rates in high-dimensional, narrow feasible regions.

Contribution

The paper proposes SIRL, a novel RL approach that dynamically adjusts imitation based on trajectory relevance, significantly enhancing performance in complex humanoid control tasks.

Findings

Achieves 120% performance improvement on HumanoidBench

Effectively focuses exploration on task-relevant regions

Results show meaningful behavior improvements and task success

Abstract

Complex high-dimensional spaces with high Degree-of-Freedom and complicated action spaces, such as humanoid robots equipped with dexterous hands, pose significant challenges for reinforcement learning (RL) algorithms, which need to wisely balance exploration and exploitation under limited sample budgets. In general, feasible regions for accomplishing tasks within complex high-dimensional spaces are exceedingly narrow. For instance, in the context of humanoid robot motion control, the vast majority of space corresponds to falling, while only a minuscule fraction corresponds to standing upright, which is conducive to the completion of downstream tasks. Once the robot explores into a potentially task-relevant region, it should place greater emphasis on the data within that region. Building on this insight, we propose the $\textbf{S}$elf-$\textbf{I}$mitative $\textbf{R}$einforcement $\textbf{L}$earning ($\textbf{SIRL}$) framework, where the RL algorithm also imitates potentially task-relevant trajectories. Specifically, trajectory return is utilized to determine its relevance to the task and an additional behavior cloning is adopted whose weight is dynamically adjusted based on the trajectory return. As a result, our proposed algorithm achieves 120% performance improvement on the challenging HumanoidBench with 5% extra computation overhead. With further visualization, we find the significant performance gain does lead to meaningful behavior improvement that several tasks are solved successfully.