A Closed-Form Geometric Retargeting Solver for Upper Body Humanoid Robot Teleoperation

TL;DR

This paper introduces SEW-Mimic, a fast, geometric retargeting method for upper body humanoid robot teleoperation that aligns robot arm orientations to human arm orientations using keypoints, offering improved speed and accuracy.

Contribution

It presents a closed-form, orientation-based retargeting algorithm that outperforms existing methods in speed and accuracy, suitable for real-time humanoid teleoperation.

Findings

Achieves 3 kHz inference speed on standard CPUs.

Outperforms other retargeting methods in accuracy and computation time.

Enhances teleoperation success and policy learning with smoother data.

Abstract

Retargeting human motion to robot poses is a practical approach for teleoperating bimanual humanoid robot arms, but existing methods can be suboptimal and slow, often causing undesirable motion or latency. This is due to optimizing to match robot end-effector to human hand position and orientation, which can also limit the robot's workspace to that of the human. Instead, this paper reframes retargeting as an orientation alignment problem, enabling a closed-form, geometric solution algorithm with an optimality guarantee. The key idea is to align a robot arm to a human's upper and lower arm orientations, as identified from shoulder, elbow, and wrist (SEW) keypoints; hence, the method is called SEW-Mimic. The method has fast inference (3 kHz) on standard commercial CPUs, leaving computational overhead for downstream applications; an example in this paper is a safety filter to avoid bimanual self-collision. The method suits most 7-degree-of-freedom robot arms and humanoids, and is agnostic to input keypoint source. Experiments show that SEW-Mimic outperforms other retargeting methods in computation time and accuracy. A pilot user study suggests that the method improves teleoperation task success. Preliminary analysis indicates that data collected with SEW-Mimic improves policy learning due to being smoother. SEW-Mimic is also shown to be a drop-in way to accelerate full-body humanoid retargeting. Finally, hardware demonstrations illustrate SEW-Mimic's practicality. The results emphasize the utility of SEW-Mimic as a fundamental building block for bimanual robot manipulation and humanoid robot teleoperation.