RT-Sketch: Goal-Conditioned Imitation Learning from Hand-Drawn Sketches

TL;DR

RT-Sketch introduces a novel goal-conditioned imitation learning approach using hand-drawn sketches, enabling robust manipulation in ambiguous or distractor-rich environments, and interprets varied sketch specificity.

Contribution

The paper proposes using hand-drawn sketches as goal representations in imitation learning, providing a flexible and spatially-aware alternative to language and images.

Findings

RT-Sketch performs comparably to image/language-conditioned agents in simple tasks.

RT-Sketch shows increased robustness with ambiguous language or visual distractors.

It can interpret sketches with different levels of detail.

Abstract

Natural language and images are commonly used as goal representations in goal-conditioned imitation learning (IL). However, natural language can be ambiguous and images can be over-specified. In this work, we propose hand-drawn sketches as a modality for goal specification in visual imitation learning. Sketches are easy for users to provide on the fly like language, but similar to images they can also help a downstream policy to be spatially-aware and even go beyond images to disambiguate task-relevant from task-irrelevant objects. We present RT-Sketch, a goal-conditioned policy for manipulation that takes a hand-drawn sketch of the desired scene as input, and outputs actions. We train RT-Sketch on a dataset of paired trajectories and corresponding synthetically generated goal sketches. We evaluate this approach on six manipulation skills involving tabletop object rearrangements on an articulated countertop. Experimentally we find that RT-Sketch is able to perform on a similar level to image or language-conditioned agents in straightforward settings, while achieving greater robustness when language goals are ambiguous or visual distractors are present. Additionally, we show that RT-Sketch has the capacity to interpret and act upon sketches with varied levels of specificity, ranging from minimal line drawings to detailed, colored drawings. For supplementary material and videos, please refer to our website: http://rt-sketch.github.io.