FlowControl: Optical Flow Based Visual Servoing

TL;DR

FlowControl is a real-time visual servoing method that uses learning-based optical flow and RGB-D data to enable one-shot imitation of manipulation tasks without needing 3D models, demonstrating robustness and generalization.

Contribution

It introduces FlowControl, a novel optical flow-based visual servoing approach that simplifies setup and enhances robustness for one-shot imitation learning in manipulation tasks.

Findings

Effective on tasks requiring precise motions

Generalizes well across different scenarios

No need for 3D object models

Abstract

One-shot imitation is the vision of robot programming from a single demonstration, rather than by tedious construction of computer code. We present a practical method for realizing one-shot imitation for manipulation tasks, exploiting modern learning-based optical flow to perform real-time visual servoing. Our approach, which we call FlowControl, continuously tracks a demonstration video, using a specified foreground mask to attend to an object of interest. Using RGB-D observations, FlowControl requires no 3D object models, and is easy to set up. FlowControl inherits great robustness to visual appearance from decades of work in optical flow. We exhibit FlowControl on a range of problems, including ones requiring very precise motions, and ones requiring the ability to generalize.