Visual Backtracking Teleoperation: A Data Collection Protocol for Offline Image-Based Reinforcement Learning

TL;DR

This paper introduces Visual Backtracking Teleoperation (VBT), a novel data collection protocol that enhances offline image-based reinforcement learning for robotic tasks by including failures and recoveries, leading to improved value functions and policies.

Contribution

The paper proposes VBT, a new data collection method that captures diverse robot behaviors, including failures and recoveries, to improve offline RL performance from small datasets.

Findings

VBT data improves value function accuracy.

Offline RL on VBT data outperforms behavior cloning by 13%.

VBT enhances policy robustness in deformable manipulation tasks.

Abstract

We consider how to most efficiently leverage teleoperator time to collect data for learning robust image-based value functions and policies for sparse reward robotic tasks. To accomplish this goal, we modify the process of data collection to include more than just successful demonstrations of the desired task. Instead we develop a novel protocol that we call Visual Backtracking Teleoperation (VBT), which deliberately collects a dataset of visually similar failures, recoveries, and successes. VBT data collection is particularly useful for efficiently learning accurate value functions from small datasets of image-based observations. We demonstrate VBT on a real robot to perform continuous control from image observations for the deformable manipulation task of T-shirt grasping. We find that by adjusting the data collection process we improve the quality of both the learned value functions and policies over a variety of baseline methods for data collection. Specifically, we find that offline reinforcement learning on VBT data outperforms standard behavior cloning on successful demonstration data by 13% when both methods are given equal-sized datasets of 60 minutes of data from the real robot.