Superevents: Towards Native Semantic Segmentation for Event-based Cameras

TL;DR

This paper introduces superevents, a novel intermediate representation for event-based cameras, enhancing tasks like semantic segmentation and tracking by leveraging locally consistent units derived through lifetime augmentation.

Contribution

The paper proposes a new method to generate superevents from event streams using lifetime augmentation, improving downstream visual tasks for event cameras.

Findings

Superevents improve semantic segmentation accuracy.

The method outperforms existing approaches on benchmark datasets.

Superevents enable better depth estimation and tracking.

Abstract

Most successful computer vision models transform low-level features, such as Gabor filter responses, into richer representations of intermediate or mid-level complexity for downstream visual tasks. These mid-level representations have not been explored for event cameras, although it is especially relevant to the visually sparse and often disjoint spatial information in the event stream. By making use of locally consistent intermediate representations, termed as superevents, numerous visual tasks ranging from semantic segmentation, visual tracking, depth estimation shall benefit. In essence, superevents are perceptually consistent local units that delineate parts of an object in a scene. Inspired by recent deep learning architectures, we present a novel method that employs lifetime augmentation for obtaining an event stream representation that is fed to a fully convolutional network to extract superevents. Our qualitative and quantitative experimental results on several sequences of a benchmark dataset highlights the significant potential for event-based downstream applications.