Language-Model-Assisted Bi-Level Programming for Reward Learning from Internet Videos

TL;DR

This paper introduces a novel bi-level programming framework that uses language models and vision-language models to learn reward functions directly from internet videos, simplifying the process of reward learning for complex behaviors.

Contribution

It presents a new framework combining VLMs and LLMs for direct reward learning from videos, bypassing complex data extraction pipelines.

Findings

Effective reward learning from YouTube videos demonstrated

Enables complex behavior synthesis with biological expert videos

Streamlines reward design process for reinforcement learning

Abstract

Learning from Demonstrations, particularly from biological experts like humans and animals, often encounters significant data acquisition challenges. While recent approaches leverage internet videos for learning, they require complex, task-specific pipelines to extract and retarget motion data for the agent. In this work, we introduce a language-model-assisted bi-level programming framework that enables a reinforcement learning agent to directly learn its reward from internet videos, bypassing dedicated data preparation. The framework includes two levels: an upper level where a vision-language model (VLM) provides feedback by comparing the learner's behavior with expert videos, and a lower level where a large language model (LLM) translates this feedback into reward updates. The VLM and LLM collaborate within this bi-level framework, using a "chain rule" approach to derive a valid search direction for reward learning. We validate the method for reward learning from YouTube videos, and the results have shown that the proposed method enables efficient reward design from expert videos of biological agents for complex behavior synthesis.