Event Masked Autoencoder: Point-wise Action Recognition with Event-Based Cameras

TL;DR

This paper introduces a novel event mask autoencoder framework that effectively captures spatiotemporal information from event-based camera data for action recognition, leveraging pre-training and transformer models to improve accuracy.

Contribution

It presents the first pre-training method for raw event camera data and proposes a new event points patch embedding for transformer-based action recognition.

Findings

Enhanced action recognition accuracy with event data.

Robustness to sensor noise and environmental factors.

Effective utilization of spatiotemporal structure in event data.

Abstract

Dynamic vision sensors (DVS) are bio-inspired devices that capture visual information in the form of asynchronous events, which encode changes in pixel intensity with high temporal resolution and low latency. These events provide rich motion cues that can be exploited for various computer vision tasks, such as action recognition. However, most existing DVS-based action recognition methods lose temporal information during data transformation or suffer from noise and outliers caused by sensor imperfections or environmental factors. To address these challenges, we propose a novel framework that preserves and exploits the spatiotemporal structure of event data for action recognition. Our framework consists of two main components: 1) a point-wise event masked autoencoder (MAE) that learns a compact and discriminative representation of event patches by reconstructing them from masked raw event camera points data; 2) an improved event points patch generation algorithm that leverages an event data inlier model and point-wise data augmentation techniques to enhance the quality and diversity of event points patches. To the best of our knowledge, our approach introduces the pre-train method into event camera raw points data for the first time, and we propose a novel event points patch embedding to utilize transformer-based models on event cameras.