SwinSF: Image Reconstruction from Spatial-Temporal Spike Streams

TL;DR

This paper introduces SwinSF, a novel deep learning model that leverages spatial-temporal attention mechanisms to improve image reconstruction from high-resolution spike camera data, setting new performance benchmarks.

Contribution

The paper presents SwinSF, a new model combining shifted window self-attention and temporal spike attention for enhanced dynamic scene reconstruction from spike streams, along with a new relevant dataset.

Findings

Achieves state-of-the-art reconstruction accuracy

Demonstrates robustness across real-world and synthetic datasets

Sets new benchmarks in spike image reconstruction

Abstract

The spike camera, with its high temporal resolution, low latency, and high dynamic range, addresses high-speed imaging challenges like motion blur. It captures photons at each pixel independently, creating binary spike streams rich in temporal information but challenging for image reconstruction. Current algorithms, both traditional and deep learning-based, still need to be improved in the utilization of the rich temporal detail and the restoration of the details of the reconstructed image. To overcome this, we introduce Swin Spikeformer (SwinSF), a novel model for dynamic scene reconstruction from spike streams. SwinSF is composed of Spike Feature Extraction, Spatial-Temporal Feature Extraction, and Final Reconstruction Module. It combines shifted window self-attention and proposed temporal spike attention, ensuring a comprehensive feature extraction that encapsulates both spatial and temporal dynamics, leading to a more robust and accurate reconstruction of spike streams. Furthermore, we build a new synthesized dataset for spike image reconstruction which matches the resolution of the latest spike camera, ensuring its relevance and applicability to the latest developments in spike camera imaging. Experimental results demonstrate that the proposed network SwinSF sets a new benchmark, achieving state-of-the-art performance across a series of datasets, including both real-world and synthesized data across various resolutions. Our codes and proposed dataset will be available soon.