User-oriented Natural Human-Robot Control with Thin-Plate Splines and LRCN

TL;DR

This paper introduces a real-time vision-based teleoperation system for robotic arms that uses a depth camera and natural user interface, employing Thin-Plate Splines for pose mapping and LRCN for hand state classification, enabling intuitive control without wearable devices.

Contribution

The paper presents a novel combination of TPS-based nonlinear pose mapping and LRCN-based hand state classification for natural, wearable-free robot teleoperation.

Findings

LRCN outperforms single image CNN in hand state classification.

Users quickly learn to operate the system effectively.

TPS mapping improves precision near workspace boundaries.

Abstract

We propose a real-time vision-based teleoperation approach for robotic arms that employs a single depth-based camera, exempting the user from the need for any wearable devices. By employing a natural user interface, this novel approach leverages the conventional fine-tuning control, turning it into a direct body pose capture process. The proposed approach is comprised of two main parts. The first is a nonlinear customizable pose mapping based on Thin-Plate Splines (TPS), to directly transfer human body motion to robotic arm motion in a nonlinear fashion, thus allowing matching dissimilar bodies with different workspace shapes and kinematic constraints. The second is a Deep Neural Network hand-state classifier based on Long-term Recurrent Convolutional Networks (LRCN) that exploits the temporal coherence of the acquired depth data. We validate, evaluate and compare our approach through both classical cross-validation experiments of the proposed hand state classifier; and user studies over a set of practical experiments involving variants of pick-and-place and manufacturing tasks. Results revealed that LRCN networks outperform single image Convolutional Neural Networks; and that users' learning curves were steep, thus allowing the successful completion of the proposed tasks. When compared to a previous approach, the TPS approach revealed no increase in task complexity and similar times of completion, while providing more precise operation in regions closer to workspace boundaries.