Rethinking High-speed Image Reconstruction Framework with Spike Camera

TL;DR

This paper introduces SpikeCLIP, a novel spike-to-image reconstruction framework that leverages CLIP's text-image alignment and unpaired datasets to improve high-speed image reconstruction from spike cameras, especially under low-light conditions.

Contribution

SpikeCLIP innovatively combines CLIP's capabilities with unpaired datasets to enhance spike camera image reconstruction, addressing noise and domain gap issues in low-light environments.

Findings

Significantly improves texture details in reconstructed images

Enhances luminance balance and visual quality

Aligns images better with downstream task requirements

Abstract

Spike cameras, as innovative neuromorphic devices, generate continuous spike streams to capture high-speed scenes with lower bandwidth and higher dynamic range than traditional RGB cameras. However, reconstructing high-quality images from the spike input under low-light conditions remains challenging. Conventional learning-based methods often rely on the synthetic dataset as the supervision for training. Still, these approaches falter when dealing with noisy spikes fired under the low-light environment, leading to further performance degradation in the real-world dataset. This phenomenon is primarily due to inadequate noise modelling and the domain gap between synthetic and real datasets, resulting in recovered images with unclear textures, excessive noise, and diminished brightness. To address these challenges, we introduce a novel spike-to-image reconstruction framework SpikeCLIP that goes beyond traditional training paradigms. Leveraging the CLIP model's powerful capability to align text and images, we incorporate the textual description of the captured scene and unpaired high-quality datasets as the supervision. Our experiments on real-world low-light datasets U-CALTECH and U-CIFAR demonstrate that SpikeCLIP significantly enhances texture details and the luminance balance of recovered images. Furthermore, the reconstructed images are well-aligned with the broader visual features needed for downstream tasks, ensuring more robust and versatile performance in challenging environments.