EventCrab: Harnessing Frame and Point Synergy for Event-based Action Recognition and Beyond

TL;DR

EventCrab introduces a synergy-aware framework that combines dense frame-based and sparse point-based networks to improve event-based action recognition by leveraging their complementary strengths.

Contribution

The paper proposes a novel framework that integrates frame-specific and point-specific networks with a joint representation space for better asynchronous event data processing.

Findings

Achieved 5.17% improvement on SeAct dataset.

Achieved 7.01% improvement on HARDVS dataset.

Demonstrated significant performance gains over existing methods.

Abstract

Event-based Action Recognition (EAR) possesses the advantages of high-temporal resolution capturing and privacy preservation compared with traditional action recognition. Current leading EAR solutions typically follow two regimes: project unconstructed event streams into dense constructed event frames and adopt powerful frame-specific networks, or employ lightweight point-specific networks to handle sparse unconstructed event points directly. However, such two regimes are blind to a fundamental issue: failing to accommodate the unique dense temporal and sparse spatial properties of asynchronous event data. In this article, we present a synergy-aware framework, i.e., EventCrab, that adeptly integrates the "lighter" frame-specific networks for dense event frames with the "heavier" point-specific networks for sparse event points, balancing accuracy and efficiency. Furthermore, we establish a joint frame-text-point representation space to bridge distinct event frames and points. In specific, to better exploit the unique spatiotemporal relationships inherent in asynchronous event points, we devise two strategies for the "heavier" point-specific embedding: i) a Spiking-like Context Learner (SCL) that extracts contextualized event points from raw event streams. ii) an Event Point Encoder (EPE) that further explores event-point long spatiotemporal features in a Hilbert-scan way. Experiments on four datasets demonstrate the significant performance of our proposed EventCrab, particularly gaining improvements of 5.17% on SeAct and 7.01% on HARDVS.