Unveiling the Potential of Spike Streams for Foreground Occlusion Removal from Densely Continuous Views

TL;DR

This paper introduces a novel spike camera-based method for removing dense foreground occlusions from scenes using continuous multi-view imaging, without prior camera calibration, achieving effective de-occlusion in real-world scenarios.

Contribution

The paper presents SpkOccNet, a new model that fuses spike stream data from continuous viewpoints with a cross-view attention mechanism, and provides the first real spike-based dataset for occlusion removal.

Findings

Efficient removal of dense occlusions in diverse scenes

Strong generalization to real-world data

No prior camera calibration needed

Abstract

The extraction of a clean background image by removing foreground occlusion holds immense practical significance, but it also presents several challenges. Presently, the majority of de-occlusion research focuses on addressing this issue through the extraction and synthesis of discrete images from calibrated camera arrays. Nonetheless, the restoration quality tends to suffer when faced with dense occlusions or high-speed motions due to limited perspectives and motion blur. To successfully remove dense foreground occlusion, an effective multi-view visual information integration approach is required. Introducing the spike camera as a novel type of neuromorphic sensor offers promising capabilities with its ultra-high temporal resolution and high dynamic range. In this paper, we propose an innovative solution for tackling the de-occlusion problem through continuous multi-view imaging using only one spike camera without any prior knowledge of camera intrinsic parameters and camera poses. By rapidly moving the spike camera, we continually capture the dense stream of spikes from the occluded scene. To process the spikes, we build a novel model \textbf{SpkOccNet}, in which we integrate information of spikes from continuous viewpoints within multi-windows, and propose a novel cross-view mutual attention mechanism for effective fusion and refinement. In addition, we contribute the first real-world spike-based dataset \textbf{S-OCC} for occlusion removal. The experimental results demonstrate that our proposed model efficiently removes dense occlusions in diverse scenes while exhibiting strong generalization.