Seeing Behind Dynamic Occlusions with Event Cameras

TL;DR

This paper introduces a novel method combining traditional and event cameras to reconstruct backgrounds behind dynamic occlusions, outperforming existing inpainting techniques by leveraging high-temporal-resolution event data.

Contribution

First to utilize event cameras with traditional cameras for background reconstruction behind dynamic occlusions, providing a new approach that handles real-time changing patterns.

Findings

Outperforms image inpainting by 3dB PSNR on the new dataset.

Provides the first large-scale dataset of synchronized images and event sequences.

Demonstrates effective background reconstruction in dynamic occlusion scenarios.

Abstract

Unwanted camera occlusions, such as debris, dust, rain-drops, and snow, can severely degrade the performance of computer-vision systems. Dynamic occlusions are particularly challenging because of the continuously changing pattern. Existing occlusion-removal methods currently use synthetic aperture imaging or image inpainting. However, they face issues with dynamic occlusions as these require multiple viewpoints or user-generated masks to hallucinate the background intensity. We propose a novel approach to reconstruct the background from a single viewpoint in the presence of dynamic occlusions. Our solution relies for the first time on the combination of a traditional camera with an event camera. When an occlusion moves across a background image, it causes intensity changes that trigger events. These events provide additional information on the relative intensity changes between foreground and background at a high temporal resolution, enabling a truer reconstruction of the background content. We present the first large-scale dataset consisting of synchronized images and event sequences to evaluate our approach. We show that our method outperforms image inpainting methods by 3dB in terms of PSNR on our dataset.