Exploring Event-based Human Pose Estimation with 3D Event Representations

TL;DR

This paper introduces novel 3D event representations, RasEPC and DEV, to improve human pose estimation from event camera data, especially in challenging high-dynamic scenarios, and provides a new synthetic dataset EV-3DPW for training and evaluation.

Contribution

The paper proposes two innovative 3D event representations, RasEPC and DEV, to enhance pose estimation accuracy and efficiency from event data, along with a new synthetic dataset EV-3DPW.

Findings

RasEPC reduces memory and computational costs while preserving 3D event information.

DEV utilizes decoupled event attention to extract 3D cues from 2D projections.

Models trained on EV-3DPW perform well on multiple datasets, demonstrating effectiveness.

Abstract

Human pose estimation is a fundamental and appealing task in computer vision. Although traditional cameras are commonly applied, their reliability decreases in scenarios under high dynamic range or heavy motion blur, where event cameras offer a robust solution. Predominant event-based methods accumulate events into frames, ignoring the asynchronous and high temporal resolution that is crucial for distinguishing distinct actions. To address this issue and to unlock the 3D potential of event information, we introduce two 3D event representations: the Rasterized Event Point Cloud (RasEPC) and the Decoupled Event Voxel (DEV). The RasEPC aggregates events within concise temporal slices at identical positions, preserving their 3D attributes along with statistical information, thereby significantly reducing memory and computational demands. Meanwhile, the DEV representation discretizes events into voxels and projects them across three orthogonal planes, utilizing decoupled event attention to retrieve 3D cues from the 2D planes. Furthermore, we develop and release EV-3DPW, a synthetic event-based dataset crafted to facilitate training and quantitative analysis in outdoor scenes. Our methods are tested on the DHP19 public dataset, MMHPSD dataset, and our EV-3DPW dataset, with further qualitative validation via a derived driving scene dataset EV-JAAD and an outdoor collection vehicle. Our code and dataset have been made publicly available at https://github.com/MasterHow/EventPointPose.