EventHands: Real-Time Neural 3D Hand Pose Estimation from an Event Stream

TL;DR

EventHands introduces a novel real-time 3D hand pose estimation method using a single event camera, achieving high temporal resolution and superior speed compared to traditional RGB or depth camera-based methods.

Contribution

This work is the first to utilize a single event camera for 3D hand pose estimation, developing a new neural approach and event stream representation for learning and generalization.

Findings

Outperforms recent monocular methods in accuracy

Captures hand motions at unprecedented speeds

Operates in real-time at 1000 Hz

Abstract

3D hand pose estimation from monocular videos is a long-standing and challenging problem, which is now seeing a strong upturn. In this work, we address it for the first time using a single event camera, i.e., an asynchronous vision sensor reacting on brightness changes. Our EventHands approach has characteristics previously not demonstrated with a single RGB or depth camera such as high temporal resolution at low data throughputs and real-time performance at 1000 Hz. Due to the different data modality of event cameras compared to classical cameras, existing methods cannot be directly applied to and re-trained for event streams. We thus design a new neural approach which accepts a new event stream representation suitable for learning, which is trained on newly-generated synthetic event streams and can generalise to real data. Experiments show that EventHands outperforms recent monocular methods using a colour (or depth) camera in terms of accuracy and its ability to capture hand motions of unprecedented speed. Our method, the event stream simulator and the dataset are publicly available; see https://4dqv.mpi-inf.mpg.de/EventHands/