Real-Time Online Skeleton Extraction and Gesture Recognition on Pepper

TL;DR

This paper introduces a real-time system on Pepper robot that combines skeleton extraction and gesture recognition using deep CNNs and fisheye camera, addressing challenges with unknown gestures in real scenarios.

Contribution

It presents the first integrated real-time system for skeleton extraction and gesture recognition on Pepper robot using embedded GPU and fisheye camera.

Findings

Successful real-time skeleton and gesture recognition on Pepper

System handles unknown human gestures effectively

Demonstrates feasibility in challenging real-world scenarios

Abstract

We present a multi-stage pipeline for simple gesture recognition. The novelty of our approach is the association of different technologies, resulting in the first real-time system as of now to conjointly extract skeletons and recognise gesture on a Pepper robot. For this task, Pepper has been augmented with an embedded GPU for running deep CNNs and a fish-eye camera to capture whole scene interaction. We show in this article that real-case scenarios are challenging, and the state-of-the-art approaches hardly deal with unknown human gestures. We present here a way to handle such cases.