Learning Sensorimotor Primitives of Sequential Manipulation Tasks from Visual Demonstrations

TL;DR

This paper introduces a neural network framework that learns both low-level and high-level manipulation policies directly from raw visual demonstrations, enabling robots to perform complex sequential tasks without manual annotations.

Contribution

It presents a novel approach to learn sensorimotor primitives for sequential manipulation tasks directly from raw videos, combining low-level control and high-level decision-making.

Findings

Efficient learning from real visual demonstrations.

Outperforms existing imitation learning algorithms.

Successfully handles complex sequential manipulation tasks.

Abstract

This work aims to learn how to perform complex robot manipulation tasks that are composed of several, consecutively executed low-level sub-tasks, given as input a few visual demonstrations of the tasks performed by a person. The sub-tasks consist of moving the robot's end-effector until it reaches a sub-goal region in the task space, performing an action, and triggering the next sub-task when a pre-condition is met. Most prior work in this domain has been concerned with learning only low-level tasks, such as hitting a ball or reaching an object and grasping it. This paper describes a new neural network-based framework for learning simultaneously low-level policies as well as high-level policies, such as deciding which object to pick next or where to place it relative to other objects in the scene. A key feature of the proposed approach is that the policies are learned directly from raw videos of task demonstrations, without any manual annotation or post-processing of the data. Empirical results on object manipulation tasks with a robotic arm show that the proposed network can efficiently learn from real visual demonstrations to perform the tasks, and outperforms popular imitation learning algorithms.