Tracking Fast by Learning Slow: An Event-based Speed Adaptive Hand Tracker Leveraging Knowledge in RGB Domain

TL;DR

This paper introduces an event-based speed adaptive hand tracker that leverages RGB knowledge, constructed a new dataset, and employs novel data augmentation and segmentation methods to improve fast hand tracking accuracy.

Contribution

The paper presents the first 3D hand tracking dataset from an event camera and proposes a CNN model with domain adaptation, data augmentation, and event stream segmentation for fast hand tracking.

Findings

Outperforms RGB-based hand tracking methods in fast motion scenarios.

Outperforms previous event-based solutions in speed and accuracy.

Demonstrates effectiveness of domain adaptation and data augmentation techniques.

Abstract

3D hand tracking methods based on monocular RGB videos are easily affected by motion blur, while event camera, a sensor with high temporal resolution and dynamic range, is naturally suitable for this task with sparse output and low power consumption. However, obtaining 3D annotations of fast-moving hands is difficult for constructing event-based hand-tracking datasets. In this paper, we provided an event-based speed adaptive hand tracker (ESAHT) to solve the hand tracking problem based on event camera. We enabled a CNN model trained on a hand tracking dataset with slow motion, which enabled the model to leverage the knowledge of RGB-based hand tracking solutions, to work on fast hand tracking tasks. To realize our solution, we constructed the first 3D hand tracking dataset captured by an event camera in a real-world environment, figured out two data augment methods to narrow the domain gap between slow and fast motion data, developed a speed adaptive event stream segmentation method to handle hand movements in different moving speeds, and introduced a new event-to-frame representation method adaptive to event streams with different lengths. Experiments showed that our solution outperformed RGB-based as well as previous event-based solutions in fast hand tracking tasks, and our codes and dataset will be publicly available.