OmniEvent: Unified Event Representation Learning

TL;DR

OmniEvent introduces a unified event representation learning framework that achieves state-of-the-art performance across multiple tasks by decoupling spatial and temporal features and efficiently fusing them.

Contribution

It is the first framework to unify event data processing, removing the need for task-specific designs and enabling standard vision models to handle event data.

Findings

Outperforms task-specific SOTA by up to 68.2% across 10 datasets

Uses a decouple-enhance-fuse paradigm for better handling of inhomogeneity

Achieves high efficiency with space-filling curves and attention-based fusion

Abstract

Event cameras have gained increasing popularity in computer vision due to their ultra-high dynamic range and temporal resolution. However, event networks heavily rely on task-specific designs due to the unstructured data distribution and spatial-temporal (S-T) inhomogeneity, making it hard to reuse existing architectures for new tasks. We propose OmniEvent, the first unified event representation learning framework that achieves SOTA performance across diverse tasks, fully removing the need of task-specific designs. Unlike previous methods that treat event data as 3D point clouds with manually tuned S-T scaling weights, OmniEvent proposes a decouple-enhance-fuse paradigm, where the local feature aggregation and enhancement is done independently on the spatial and temporal domains to avoid inhomogeneity issues. Space-filling curves are applied to enable large receptive fields while improving memory and compute efficiency. The features from individual domains are then fused by attention to learn S-T interactions. The output of OmniEvent is a grid-shaped tensor, which enables standard vision models to process event data without architecture change. With a unified framework and similar hyper-parameters, OmniEvent out-performs (tasks-specific) SOTA by up to 68.2% across 3 representative tasks and 10 datasets (Fig.1). Code will be ready in https://github.com/Wickyan/OmniEvent .