EventGAN: Leveraging Large Scale Image Datasets for Event Cameras

TL;DR

This paper introduces EventGAN, a generative model that simulates event camera data from images, enabling the training of vision models for event data without requiring real event labels.

Contribution

EventGAN leverages existing labeled image datasets to generate realistic event data, facilitating the development of deep learning models for event-based vision tasks.

Findings

EventGAN can generate high-quality event data from images.

Models trained on simulated event data generalize well to real event datasets.

The approach reduces the need for extensive labeled event data.

Abstract

Event cameras provide a number of benefits over traditional cameras, such as the ability to track incredibly fast motions, high dynamic range, and low power consumption. However, their application into computer vision problems, many of which are primarily dominated by deep learning solutions, has been limited by the lack of labeled training data for events. In this work, we propose a method which leverages the existing labeled data for images by simulating events from a pair of temporal image frames, using a convolutional neural network. We train this network on pairs of images and events, using an adversarial discriminator loss and a pair of cycle consistency losses. The cycle consistency losses utilize a pair of pre-trained self-supervised networks which perform optical flow estimation and image reconstruction from events, and constrain our network to generate events which result in accurate outputs from both of these networks. Trained fully end to end, our network learns a generative model for events from images without the need for accurate modeling of the motion in the scene, exhibited by modeling based methods, while also implicitly modeling event noise. Using this simulator, we train a pair of downstream networks on object detection and 2D human pose estimation from events, using simulated data from large scale image datasets, and demonstrate the networks' abilities to generalize to datasets with real events.