ExtremControl: Low-Latency Humanoid Teleoperation with Direct Extremity Control

TL;DR

ExtremControl is a low-latency humanoid teleoperation framework that directly controls extremities in SE(3) space, enabling highly responsive behaviors with latency as low as 50ms.

Contribution

It introduces a novel control approach operating directly on extremity poses, reducing latency and improving responsiveness over prior methods.

Findings

Achieves end-to-end latency as low as 50ms in real-world tests.

Supports dynamic tasks like juggling and ping-pong with high responsiveness.

System surpasses previous latency limits of 200ms.

Abstract

Building a low-latency humanoid teleoperation system is essential for collecting diverse reactive and dynamic demonstrations. However, existing approaches rely on heavily pre-processed human-to-humanoid motion retargeting and position-only PD control, resulting in substantial latency that severely limits responsiveness and prevents tasks requiring rapid feedback and fast reactions. To address this problem, we propose ExtremControl, a low latency whole-body control framework that: (1) operates directly on SE(3) poses of selected rigid links, primarily humanoid extremities, to avoid full-body retargeting; (2) utilizes a Cartesian-space mapping to directly convert human motion to humanoid link targets; and (3) incorporates velocity feedforward control at low level to support highly responsive behavior under rapidly changing control interfaces. We further provide a unified theoretical formulation of ExtremControl and systematically validate its effectiveness through experiments in both simulation and real-world environments. Building on ExtremControl, we implement a low-latency humanoid teleoperation system that supports both optical motion capture and VR-based motion tracking, achieving end-to-end latency as low as 50ms and enabling highly responsive behaviors such as ping-pong ball balancing, juggling, and real-time return, thereby substantially surpassing the 200ms latency limit observed in prior work.