EventEgoHands: Event-based Egocentric 3D Hand Mesh Reconstruction

TL;DR

EventEgoHands is a new method for reconstructing 3D hand meshes from event camera data in egocentric views, overcoming background noise issues and showing significant accuracy improvements in dynamic environments.

Contribution

It introduces a Hand Segmentation Module to reduce background noise effects, enabling effective 3D hand mesh reconstruction from event data in egocentric scenarios.

Findings

Improves MPJPE by over 4.5 cm (43%) on N-HOT3D dataset.

Effectively mitigates background noise influence in event-based hand reconstruction.

Demonstrates robustness in dynamic, egocentric environments.

Abstract

Reconstructing 3D hand mesh is challenging but an important task for human-computer interaction and AR/VR applications. In particular, RGB and/or depth cameras have been widely used in this task. However, methods using these conventional cameras face challenges in low-light environments and during motion blur. Thus, to address these limitations, event cameras have been attracting attention in recent years for their high dynamic range and high temporal resolution. Despite their advantages, event cameras are sensitive to background noise or camera motion, which has limited existing studies to static backgrounds and fixed cameras. In this study, we propose EventEgoHands, a novel method for event-based 3D hand mesh reconstruction in an egocentric view. Our approach introduces a Hand Segmentation Module that extracts hand regions, effectively mitigating the influence of dynamic background events. We evaluated our approach and demonstrated its effectiveness on the N-HOT3D dataset, improving MPJPE by approximately more than 4.5 cm (43%).