Generalization in Dexterous Manipulation via Geometry-Aware Multi-Task Learning

TL;DR

This paper demonstrates that reinforcement learning combined with multi-task training and geometry-aware object representations enables a single policy to generalize across over 100 diverse objects for in-hand manipulation, outperforming specialized policies.

Contribution

It introduces a multi-task learning approach with object point cloud representations that enhances generalization in dexterous manipulation tasks.

Findings

A single policy can manipulate over 100 diverse objects.

Multi-task learning outperforms single-object policies on seen and unseen objects.

Object point cloud representations improve generalization and performance.

Abstract

Dexterous manipulation of arbitrary objects, a fundamental daily task for humans, has been a grand challenge for autonomous robotic systems. Although data-driven approaches using reinforcement learning can develop specialist policies that discover behaviors to control a single object, they often exhibit poor generalization to unseen ones. In this work, we show that policies learned by existing reinforcement learning algorithms can in fact be generalist when combined with multi-task learning and a well-chosen object representation. We show that a single generalist policy can perform in-hand manipulation of over 100 geometrically-diverse real-world objects and generalize to new objects with unseen shape or size. Interestingly, we find that multi-task learning with object point cloud representations not only generalizes better but even outperforms the single-object specialist policies on both training as well as held-out test objects. Video results at https://huangwl18.github.io/geometry-dex