Neural Network Based Lidar Gesture Recognition for Realtime Robot Teleoperation

TL;DR

This paper introduces a lightweight, robust lidar-based gesture recognition system for real-time robot teleoperation, combining pose estimation and gesture classification modules that operate effectively outdoors with minimal data labeling.

Contribution

The paper presents a novel modular lidar-based gesture recognition system that is low-complexity, robust to outdoor conditions, and requires minimal manual data labeling.

Findings

Effective in outdoor environments

Operates with limited computational resources

Achieves accurate gesture recognition in real-world tests

Abstract

We propose a novel low-complexity lidar gesture recognition system for mobile robot control robust to gesture variation. Our system uses a modular approach, consisting of a pose estimation module and a gesture classifier. Pose estimates are predicted from lidar scans using a Convolutional Neural Network trained using an existing stereo-based pose estimation system. Gesture classification is accomplished using a Long Short-Term Memory network and uses a sequence of estimated body poses as input to predict a gesture. Breaking down the pipeline into two modules reduces the dimensionality of the input, which could be lidar scans, stereo imagery, or any other modality from which body keypoints can be extracted, making our system lightweight and suitable for mobile robot control with limited computing power. The use of lidar contributes to the robustness of the system, allowing it to operate in most outdoor conditions, to be independent of lighting conditions, and for input to be detected 360 degrees around the robot. The lidar-based pose estimator and gesture classifier use data augmentation and automated labeling techniques, requiring a minimal amount of data collection and avoiding the need for manual labeling. We report experimental results for each module of our system and demonstrate its effectiveness by testing it in a real-world robot teleoperation setting.