DexTwist: Dexterous Hand Retargeting for Twist Motion via Mixed Reality-based Teleoperation

TL;DR

DexTwist introduces a novel MR-based teleoperation framework that accurately retargets rotational hand motions to dexterous robots, overcoming embodiment gaps and improving stability in contact-rich tasks.

Contribution

It presents DexTwist, a functional twist-retargeting method that estimates screw axes and refines robot motion in real-time, enhancing dexterous manipulation accuracy.

Findings

DexTwist improves turning angle tracking over baseline methods.

It enhances screw axis stability during dexterous tasks.

Real-world experiments validate its effectiveness in contact-rich manipulation.

Abstract

Dexterous teleoperation via Mixed Reality (MR)-based interfaces offers a scalable paradigm for transferring human manipulation skills to dexterous robot hands. However, conventional retargeting approaches that minimize kinematic dissimilarity (e.g., joint angle or fingertip position error) often fail in contact-rich rotational manipulation, such as cap opening, key turning, and bolt screwing. This failure stems from the embodiment gap: mismatched link lengths, joint axes/limits, and fingertip geometry can cause direct pose imitation to induce tangential fingertip sliding rather than stable object rotation, resulting in screw axis drift, contact slip, and grasp instability. To address this, we propose DexTwist, a functional twist-retargeting framework for MR-based dexterous teleoperation. DexTwist detects a tripod pinch, estimates the operator's intended screw axis and twist magnitude, and applies a real-time residual joint-space refinement that tracks turning progress while regularizing the robot tripod geometry. The refinement minimizes a virtual-object objective defined by turning angle, screw axis consistency, fingertip closure, and tripod stability. Simulation and real-world experiments show that DexTwist improves turning angle tracking and screw axis stability compared with a vector-based retargeting baseline.