Anytime, Anywhere: Human Arm Pose from Smartwatch Data for Ubiquitous Robot Control and Teleoperation

TL;DR

This paper presents a machine learning method for estimating human arm poses from smartwatch data, enabling intuitive robot control and teleoperation with high accuracy and uncertainty measurement.

Contribution

It introduces a multimodal pose estimation approach from a single smartwatch, integrating uncertainty quantification for improved robot interaction and control.

Findings

40% reduction in prediction error over state-of-the-art

Mean error of 2.56cm for wrist and elbow positions

Enables swift, intuitive robot control and training via smartwatch

Abstract

This work devises an optimized machine learning approach for human arm pose estimation from a single smartwatch. Our approach results in a distribution of possible wrist and elbow positions, which allows for a measure of uncertainty and the detection of multiple possible arm posture solutions, i.e., multimodal pose distributions. Combining estimated arm postures with speech recognition, we turn the smartwatch into a ubiquitous, low-cost and versatile robot control interface. We demonstrate in two use-cases that this intuitive control interface enables users to swiftly intervene in robot behavior, to temporarily adjust their goal, or to train completely new control policies by imitation. Extensive experiments show that the approach results in a 40% reduction in prediction error over the current state-of-the-art and achieves a mean error of 2.56cm for wrist and elbow positions. The code is available at https://github.com/wearable-motion-capture.