Event-Stream Super Resolution using Sigma-Delta Neural Network

TL;DR

This paper presents a Sigma-Delta Neural Network-based method for super-resolving event camera data, significantly improving computational efficiency and accuracy in capturing dynamic visual scenes.

Contribution

It introduces a novel SDNN-based approach that effectively learns spatial-temporal distributions of event streams, optimized for event camera data structure.

Findings

Achieves 17.04-fold improvement in event sparsity

Realizes 32.28-fold increase in synaptic operation efficiency

Outperforms existing methods in accuracy and efficiency

Abstract

This study introduces a novel approach to enhance the spatial-temporal resolution of time-event pixels based on luminance changes captured by event cameras. These cameras present unique challenges due to their low resolution and the sparse, asynchronous nature of the data they collect. Current event super-resolution algorithms are not fully optimized for the distinct data structure produced by event cameras, resulting in inefficiencies in capturing the full dynamism and detail of visual scenes with improved computational complexity. To bridge this gap, our research proposes a method that integrates binary spikes with Sigma Delta Neural Networks (SDNNs), leveraging spatiotemporal constraint learning mechanism designed to simultaneously learn the spatial and temporal distributions of the event stream. The proposed network is evaluated using widely recognized benchmark datasets, including N-MNIST, CIFAR10-DVS, ASL-DVS, and Event-NFS. A comprehensive evaluation framework is employed, assessing both the accuracy, through root mean square error (RMSE), and the computational efficiency of our model. The findings demonstrate significant improvements over existing state-of-the-art methods, specifically, the proposed method outperforms state-of-the-art performance in computational efficiency, achieving a 17.04-fold improvement in event sparsity and a 32.28-fold increase in synaptic operation efficiency over traditional artificial neural networks, alongside a two-fold better performance over spiking neural networks.