Event Transformer. A sparse-aware solution for efficient event data processing

TL;DR

Event Transformer (EvT) is a novel, efficient, and accurate framework for event data processing that leverages event-data properties with a patch-based representation and transformer architecture, suitable for low-resource environments.

Contribution

The paper introduces a new patch-based event representation and a compact transformer architecture, achieving high efficiency and accuracy for event data processing.

Findings

EvT achieves comparable or better accuracy than state-of-the-art methods.

EvT requires significantly less computation resources.

EvT operates with minimal latency on GPU and CPU.

Abstract

Event cameras are sensors of great interest for many applications that run in low-resource and challenging environments. They log sparse illumination changes with high temporal resolution and high dynamic range, while they present minimal power consumption. However, top-performing methods often ignore specific event-data properties, leading to the development of generic but computationally expensive algorithms. Efforts toward efficient solutions usually do not achieve top-accuracy results for complex tasks. This work proposes a novel framework, Event Transformer (EvT), that effectively takes advantage of event-data properties to be highly efficient and accurate. We introduce a new patch-based event representation and a compact transformer-like architecture to process it. EvT is evaluated on different event-based benchmarks for action and gesture recognition. Evaluation results show better or comparable accuracy to the state-of-the-art while requiring significantly less computation resources, which makes EvT able to work with minimal latency both on GPU and CPU.