Naturalizing Neuromorphic Vision Event Streams Using GANs

TL;DR

This paper uses GANs, specifically Pix2Pix, to convert neuromorphic event streams into natural images, improving interpretability and classification accuracy for high-temporal-resolution vision sensors in resource-constrained environments.

Contribution

It demonstrates the application of GANs to naturalize neuromorphic event streams, enhancing interpretability and classification performance compared to raw event data.

Findings

Naturalized event streams achieve classification accuracy within 2.81% of raw images.

Improved classification accuracy by 13.19% over unprocessed event streams.

Method applied to CIFAR-10 and Linnaeus 5 datasets.

Abstract

Dynamic vision sensors are able to operate at high temporal resolutions within resource constrained environments, though at the expense of capturing static content. The sparse nature of event streams enables efficient downstream processing tasks as they are suited for power-efficient spiking neural networks. One of the challenges associated with neuromorphic vision is the lack of interpretability of event streams. While most application use-cases do not intend for the event stream to be visually interpreted by anything other than a classification network, there is a lost opportunity to integrating these sensors in spaces that conventional high-speed CMOS sensors cannot go. For example, biologically invasive sensors such as endoscopes must fit within stringent power budgets, which do not allow MHz-speeds of image integration. While dynamic vision sensing can fill this void, the interpretation challenge remains and will degrade confidence in clinical diagnostics. The use of generative adversarial networks presents a possible solution to overcoming and compensating for a vision chip's poor spatial resolution and lack of interpretability. In this paper, we methodically apply the Pix2Pix network to naturalize the event stream from spike-converted CIFAR-10 and Linnaeus 5 datasets. The quality of the network is benchmarked by performing image classification of naturalized event streams, which converges to within 2.81% of equivalent raw images, and an associated improvement over unprocessed event streams by 13.19% for the CIFAR-10 and Linnaeus 5 datasets.