Multi-finger Manipulation via Trajectory Optimization with Differentiable Rolling and Geometric Constraints

TL;DR

This paper introduces a differentiable trajectory optimization method for multi-finger dexterous manipulation that models complex geometries and finger rolling, enabling more natural and effective manipulation behaviors.

Contribution

It extends existing methods by explicitly modeling non-trivial geometries and finger rolling in a differentiable framework, improving dexterous manipulation capabilities.

Findings

Outperforms baselines in manipulation benchmarks

Successfully applies to real-world tasks

Enables emergence of finger-rolling behaviors

Abstract

Parameterizing finger rolling and finger-object contacts in a differentiable manner is important for formulating dexterous manipulation as a trajectory optimization problem. In contrast to previous methods which often assume simplified geometries of the robot and object or do not explicitly model finger rolling, we propose a method to further extend the capabilities of dexterous manipulation by accounting for non-trivial geometries of both the robot and the object. By integrating the object's Signed Distance Field (SDF) with a sampling method, our method estimates contact and rolling-related variables in a differentiable manner and includes those in a trajectory optimization framework. This formulation naturally allows for the emergence of finger-rolling behaviors, enabling the robot to locally adjust the contact points. To evaluate our method, we introduce a benchmark featuring challenging multi-finger dexterous manipulation tasks, such as screwdriver turning and in-hand reorientation. Our method outperforms baselines in terms of achieving desired object configurations and avoiding dropping the object. We also successfully apply our method to a real-world screwdriver turning task and a cuboid alignment task, demonstrating its robustness to the sim2real gap.