Learning to Manipulate Tools by Aligning Simulation to Video Demonstration

TL;DR

This paper presents an automated method for robots to learn tool manipulation by aligning simulation with real-world videos, enabling imitation learning without expert demonstrations, and demonstrating effectiveness on various tools in simulation and real robot.

Contribution

It introduces a novel alignment procedure to match simulation with video scenes and combines reinforcement learning with optimization for tool trajectory imitation.

Findings

Effective tool manipulation policies learned in simulation.

Successful transfer of policies to real robot for spade tool.

Alignment procedure improves imitation accuracy.

Abstract

A seamless integration of robots into human environments requires robots to learn how to use existing human tools. Current approaches for learning tool manipulation skills mostly rely on expert demonstrations provided in the target robot environment, for example, by manually guiding the robot manipulator or by teleoperation. In this work, we introduce an automated approach that replaces an expert demonstration with a Youtube video for learning a tool manipulation strategy. The main contributions are twofold. First, we design an alignment procedure that aligns the simulated environment with the real-world scene observed in the video. This is formulated as an optimization problem that finds a spatial alignment of the tool trajectory to maximize the sparse goal reward given by the environment. Second, we describe an imitation learning approach that focuses on the trajectory of the tool rather than the motion of the human. For this we combine reinforcement learning with an optimization procedure to find a control policy and the placement of the robot based on the tool motion in the aligned environment. We demonstrate the proposed approach on spade, scythe and hammer tools in simulation, and show the effectiveness of the trained policy for the spade on a real Franka Emika Panda robot demonstration.