Neuromorphic High-Frequency 3D Dancing Pose Estimation in Dynamic Environment

TL;DR

This paper introduces YeLan, a novel event camera-based system for high-frequency 3D human pose estimation in dynamic environments, overcoming limitations of traditional RGB-based methods in low-light and fast-motion scenarios.

Contribution

The paper presents the first dance dataset captured with event cameras and a physics-aware simulator, advancing pose estimation in challenging conditions.

Findings

YeLan outperforms baseline models in low-light and dynamic backgrounds.

Robustness demonstrated across clothing, occlusion, and lighting variations.

First dataset and simulator for event camera dance pose estimation.

Abstract

As a beloved sport worldwide, dancing is getting integrated into traditional and virtual reality-based gaming platforms nowadays. It opens up new opportunities in the technology-mediated dancing space. These platforms primarily rely on passive and continuous human pose estimation as an input capture mechanism. Existing solutions are mainly based on RGB or RGB-Depth cameras for dance games. The former suffers in low-lighting conditions due to the motion blur and low sensitivity, while the latter is too power-hungry, has a low frame rate, and has limited working distance. With ultra-low latency, energy efficiency, and wide dynamic range characteristics, the event camera is a promising solution to overcome these shortcomings. We propose YeLan, an event camera-based 3-dimensional high-frequency human pose estimation(HPE) system that survives low-lighting conditions and dynamic backgrounds. We collected the world's first event camera dance dataset and developed a fully customizable motion-to-event physics-aware simulator. YeLan outperforms the baseline models in these challenging conditions and demonstrated robustness against different types of clothing, background motion, viewing angle, occlusion, and lighting fluctuations.