EvRainDrop: HyperGraph-guided Completion for Effective Frame and Event Stream Aggregation

TL;DR

EvRainDrop introduces a hypergraph-guided completion method for event streams from event cameras, effectively addressing spatial sparsity and enabling multi-modal data integration for improved classification performance.

Contribution

This paper presents a novel hypergraph-based framework for completing and fusing multi-modal event streams, enhancing event representation learning.

Findings

Effective completion of sparse event data demonstrated

Improved classification accuracy on multiple benchmarks

Flexible integration of RGB and event data enabled

Abstract

Event cameras produce asynchronous event streams that are spatially sparse yet temporally dense. Mainstream event representation learning algorithms typically use event frames, voxels, or tensors as input. Although these approaches have achieved notable progress, they struggle to address the undersampling problem caused by spatial sparsity. In this paper, we propose a novel hypergraph-guided spatio-temporal event stream completion mechanism, which connects event tokens across different times and spatial locations via hypergraphs and leverages contextual information message passing to complete these sparse events. The proposed method can flexibly incorporate RGB tokens as nodes in the hypergraph within this completion framework, enabling multi-modal hypergraph-based information completion. Subsequently, we aggregate hypergraph node information across different time steps through self-attention, enabling effective learning and fusion of multi-modal features. Extensive experiments on both single- and multi-label event classification tasks fully validated the effectiveness of our proposed framework. The source code of this paper will be released on https://github.com/Event-AHU/EvRainDrop.