How Asynchronous Events Encode Video

TL;DR

This paper models event-based video sensors using time encoding machines, revealing how spatial sensor density influences both spatial and temporal resolution, offering advantages over traditional frame-based video.

Contribution

It introduces a theoretical framework linking sensor density to resolution in event-based video using time encoding models, highlighting benefits of oversampling in space.

Findings

Spatial sensor density affects both spatial and temporal resolution.

Oversampling in space can improve time resolution in event-based video.

Event-based sensors can emit fewer events while maintaining high resolution.

Abstract

As event-based sensing gains in popularity, theoretical understanding is needed to harness this technology's potential. Instead of recording video by capturing frames, event-based cameras have sensors that emit events when their inputs change, thus encoding information in the timing of events. This creates new challenges in establishing reconstruction guarantees and algorithms, but also provides advantages over frame-based video. We use time encoding machines to model event-based sensors: TEMs also encode their inputs by emitting events characterized by their timing and reconstruction from time encodings is well understood. We consider the case of time encoding bandlimited video and demonstrate a dependence between spatial sensor density and overall spatial and temporal resolution. Such a dependence does not occur in frame-based video, where temporal resolution depends solely on the frame rate of the video and spatial resolution depends solely on the pixel grid. However, this dependence arises naturally in event-based video and allows oversampling in space to provide better time resolution. As such, event-based vision encourages using more sensors that emit fewer events over time.