Learning Optical Flow from Event Camera with Rendered Dataset

TL;DR

This paper introduces a novel method for generating high-quality, physically accurate event-flow datasets using computer graphics, which improves the training and performance of event-based optical flow estimation models.

Contribution

We propose a rendering-based approach to create dense, accurate event-flow datasets with adjustable density, enhancing the training process for optical flow estimation from event cameras.

Findings

Rendered dataset improves optical flow learning.

Adaptive density module enhances performance.

Training on our dataset yields significant accuracy gains.

Abstract

We study the problem of estimating optical flow from event cameras. One important issue is how to build a high-quality event-flow dataset with accurate event values and flow labels. Previous datasets are created by either capturing real scenes by event cameras or synthesizing from images with pasted foreground objects. The former case can produce real event values but with calculated flow labels, which are sparse and inaccurate. The later case can generate dense flow labels but the interpolated events are prone to errors. In this work, we propose to render a physically correct event-flow dataset using computer graphics models. In particular, we first create indoor and outdoor 3D scenes by Blender with rich scene content variations. Second, diverse camera motions are included for the virtual capturing, producing images and accurate flow labels. Third, we render high-framerate videos between images for accurate events. The rendered dataset can adjust the density of events, based on which we further introduce an adaptive density module (ADM). Experiments show that our proposed dataset can facilitate event-flow learning, whereas previous approaches when trained on our dataset can improve their performances constantly by a relatively large margin. In addition, event-flow pipelines when equipped with our ADM can further improve performances.