Real-time Dexterous Telemanipulation with an End-Effect-Oriented Learning-based Approach

TL;DR

This paper introduces EFOLD, a learning-based framework for real-time dexterous telemanipulation that models interactions as a Markov Game, enabling accurate, low-latency control of robotic hands based on end-effect features.

Contribution

EFOLD is the first framework to model telemanipulation as a Markov Game with end-effect features, improving control accuracy and responsiveness.

Findings

Achieved real-time control with delay less than 0.11 seconds.

Attained high tracking accuracy with MSE below 0.084 radians.

Validated with real human subjects on virtual robotic hand tasks.

Abstract

Dexterous telemanipulation is crucial in advancing human-robot systems, especially in tasks requiring precise and safe manipulation. However, it faces significant challenges due to the physical differences between human and robotic hands, the dynamic interaction with objects, and the indirect control and perception of the remote environment. Current approaches predominantly focus on mapping the human hand onto robotic counterparts to replicate motions, which exhibits a critical oversight: it often neglects the physical interaction with objects and relegates the interaction burden to the human to adapt and make laborious adjustments in response to the indirect and counter-intuitive observation of the remote environment. This work develops an End-Effects-Oriented Learning-based Dexterous Telemanipulation (EFOLD) framework to address telemanipulation tasks. EFOLD models telemanipulation as a Markov Game, introducing multiple end-effect features to interpret the human operator's commands during interaction with objects. These features are used by a Deep Reinforcement Learning policy to control the robot and reproduce such end effects. EFOLD was evaluated with real human subjects and two end-effect extraction methods for controlling a virtual Shadow Robot Hand in telemanipulation tasks. EFOLD achieved real-time control capability with low command following latency (delay<0.11s) and highly accurate tracking (MSE<0.084 rad).