TeleDex: Accessible Dexterous Teleoperation

TL;DR

TeleDex is an open-source, phone-based teleoperation system that simplifies dexterous robot manipulation by providing low-latency control and easy demonstration collection, enhancing policy generalization in real-world tasks.

Contribution

The paper introduces TeleDex, a novel, accessible teleoperation system using smartphones for dexterous manipulation, reducing hardware barriers and enabling quick demonstration collection.

Findings

Effective in real-world manipulation tasks

Supports both handheld and hand-mounted interfaces

Facilitates policy fine-tuning through demonstration collection

Abstract

Despite increasing dataset scale and model capacity, robot manipulation policies still struggle to generalize beyond their training distributions. As a result, deploying state-of-the-art policies in new environments, tasks, or robot embodiments often requires collecting additional demonstrations. Enabling this in real-world deployment settings requires tools that allow users to collect demonstrations quickly, affordably, and with minimal setup. We present TeleDex, an open-source system for intuitive teleoperation of dexterous hands and robotic manipulators using any readily available phone. The system streams low-latency 6-DoF wrist poses and articulated 21-DoF hand state estimates from the phone, which are retargeted to robot arms and multi-fingered hands without requiring external tracking infrastructure. TeleDex supports both a handheld phone-only mode and an optional 3D-printable hand-mounted interface for finger-level teleoperation. By lowering the hardware and setup barriers to dexterous teleoperation, TeleDex enables users to quickly collect demonstrations during deployment to support policy fine-tuning. We evaluate the system across simulation and real-world manipulation tasks, demonstrating its effectiveness as a unified scalable interface for robot teleoperation. All software and hardware designs, along with demonstration videos, are open-source and available at orayyan.com/teledex.