Learning to Sequence and Blend Robot Skills via Differentiable Optimization

TL;DR

This paper presents a new framework that learns to sequence and blend robot skills seamlessly using differentiable optimization, enabling smooth multi-skill execution from demonstrations across various robotic tasks.

Contribution

It introduces a skill-agnostic approach encoding skill sequences as quadratic programs and learning from demonstrations through differentiable optimization layers.

Findings

Successful application in pick-and-place with planar robots

Effective pouring with a humanoid robot

Bimanual sweeping with a human model

Abstract

In contrast to humans and animals who naturally execute seamless motions, learning and smoothly executing sequences of actions remains a challenge in robotics. This paper introduces a novel skill-agnostic framework that learns to sequence and blend skills based on differentiable optimization. Our approach encodes sequences of previously-defined skills as quadratic programs (QP), whose parameters determine the relative importance of skills along the task. Seamless skill sequences are then learned from demonstrations by exploiting differentiable optimization layers and a tailored loss formulated from the QP optimality conditions. Via the use of differentiable optimization, our work offers novel perspectives on multitask control. We validate our approach in a pick-and-place scenario with planar robots, a pouring experiment with a real humanoid robot, and a bimanual sweeping task with a human model.