Video-Enhanced Offline Reinforcement Learning: A Model-Based Approach

TL;DR

VeoRL is a novel model-based offline reinforcement learning method that constructs an interactive world model from online video data, significantly improving policy performance in various visual control tasks.

Contribution

It introduces a new approach to offline RL by leveraging unlabeled videos to build world models, transferring knowledge to enhance policy learning.

Findings

Achieves over 100% performance improvement in some tasks

Effective in robotic manipulation, autonomous driving, and video games

Utilizes diverse online videos for world model construction

Abstract

Offline reinforcement learning (RL) enables policy optimization using static datasets, avoiding the risks and costs of extensive real-world exploration. However, it struggles with suboptimal offline behaviors and inaccurate value estimation due to the lack of environmental interaction. We present Video-Enhanced Offline RL (VeoRL), a model-based method that constructs an interactive world model from diverse, unlabeled video data readily available online. Leveraging model-based behavior guidance, our approach transfers commonsense knowledge of control policy and physical dynamics from natural videos to the RL agent within the target domain. VeoRL achieves substantial performance gains (over 100% in some cases) across visual control tasks in robotic manipulation, autonomous driving, and open-world video games.