Video2Reward: Generating Reward Function from Videos for Legged Robot Behavior Learning

TL;DR

This paper introduces a novel video2reward approach that generates reward functions directly from videos of behaviors, enabling more controllable and efficient learning of diverse legged robot motions, outperforming existing LLM-based methods.

Contribution

The paper presents a new method that converts videos into reward functions for robot learning, incorporating an iterative refinement scheme for improved accuracy and behavior diversity.

Findings

Outperforms state-of-the-art LLM-based reward methods by over 37.6% in human normalized score.

Enables rapid learning of diverse behaviors like walking and running.

Demonstrates effectiveness on both bipedal and quadrupedal robot tasks.

Abstract

Learning behavior in legged robots presents a significant challenge due to its inherent instability and complex constraints. Recent research has proposed the use of a large language model (LLM) to generate reward functions in reinforcement learning, thereby replacing the need for manually designed rewards by experts. However, this approach, which relies on textual descriptions to define learning objectives, fails to achieve controllable and precise behavior learning with clear directionality. In this paper, we introduce a new video2reward method, which directly generates reward functions from videos depicting the behaviors to be mimicked and learned. Specifically, we first process videos containing the target behaviors, converting the motion information of individuals in the videos into keypoint trajectories represented as coordinates through a video2text transforming module. These trajectories are then fed into an LLM to generate the reward function, which in turn is used to train the policy. To enhance the quality of the reward function, we develop a video-assisted iterative reward refinement scheme that visually assesses the learned behaviors and provides textual feedback to the LLM. This feedback guides the LLM to continually refine the reward function, ultimately facilitating more efficient behavior learning. Experimental results on tasks involving bipedal and quadrupedal robot motion control demonstrate that our method surpasses the performance of state-of-the-art LLM-based reward generation methods by over 37.6% in terms of human normalized score. More importantly, by switching video inputs, we find our method can rapidly learn diverse motion behaviors such as walking and running.