V2CE: Video to Continuous Events Simulator

TL;DR

This paper introduces V2CE, a novel method for converting videos into continuous event streams suitable for Dynamic Vision Sensors, addressing domain shift issues and improving event timestamp accuracy with state-of-the-art results.

Contribution

The paper presents a new video-to-events conversion method tailored for DVS, incorporating specialized losses and a dynamic-aware timestamp inference to enhance event quality and temporal accuracy.

Findings

Significantly improved event voxel quality.

Accurate continuous timestamp recovery.

Achieved state-of-the-art validation metrics.

Abstract

Dynamic Vision Sensor (DVS)-based solutions have recently garnered significant interest across various computer vision tasks, offering notable benefits in terms of dynamic range, temporal resolution, and inference speed. However, as a relatively nascent vision sensor compared to Active Pixel Sensor (APS) devices such as RGB cameras, DVS suffers from a dearth of ample labeled datasets. Prior efforts to convert APS data into events often grapple with issues such as a considerable domain shift from real events, the absence of quantified validation, and layering problems within the time axis. In this paper, we present a novel method for video-to-events stream conversion from multiple perspectives, considering the specific characteristics of DVS. A series of carefully designed losses helps enhance the quality of generated event voxels significantly. We also propose a novel local dynamic-aware timestamp inference strategy to accurately recover event timestamps from event voxels in a continuous fashion and eliminate the temporal layering problem. Results from rigorous validation through quantified metrics at all stages of the pipeline establish our method unquestionably as the current state-of-the-art (SOTA).