Text-to-Events: Synthetic Event Camera Streams from Conditional Text Input

TL;DR

This paper introduces a novel text-to-events model that synthesizes realistic event camera streams from text prompts, facilitating the creation of large labeled datasets for training deep learning models in event-based vision tasks.

Contribution

The work presents a new method combining autoencoders and diffusion models to generate synthetic event streams directly from text, addressing data scarcity in event camera research.

Findings

Generated event sequences are realistic and can be used for training classifiers.

Classification accuracy on synthetic data ranges from 42% to 92%.

The model effectively produces diverse event streams from various text prompts.

Abstract

Event cameras are advantageous for tasks that require vision sensors with low-latency and sparse output responses. However, the development of deep network algorithms using event cameras has been slow because of the lack of large labelled event camera datasets for network training. This paper reports a method for creating new labelled event datasets by using a text-to-X model, where X is one or multiple output modalities, in the case of this work, events. Our proposed text-to-events model produces synthetic event frames directly from text prompts. It uses an autoencoder which is trained to produce sparse event frames representing event camera outputs. By combining the pretrained autoencoder with a diffusion model architecture, the new text-to-events model is able to generate smooth synthetic event streams of moving objects. The autoencoder was first trained on an event camera dataset of diverse scenes. In the combined training with the diffusion model, the DVS gesture dataset was used. We demonstrate that the model can generate realistic event sequences of human gestures prompted by different text statements. The classification accuracy of the generated sequences, using a classifier trained on the real dataset, ranges between 42% to 92%, depending on the gesture group. The results demonstrate the capability of this method in synthesizing event datasets.