Learning to See Through with Events

TL;DR

This paper introduces an event-based synthetic aperture imaging method that leverages asynchronous event data to effectively see through dense occlusions and under extreme lighting, outperforming traditional approaches.

Contribution

The novel E-SAI approach combines event refocusing and a hybrid SNN-CNN network to enhance imaging in challenging conditions, advancing the state-of-the-art in event-based imaging.

Findings

Achieves high-quality imaging through dense occlusions

Performs well under extreme lighting conditions

Outperforms existing methods in qualitative and quantitative evaluations

Abstract

Although synthetic aperture imaging (SAI) can achieve the seeing-through effect by blurring out off-focus foreground occlusions while recovering in-focus occluded scenes from multi-view images, its performance is often deteriorated by dense occlusions and extreme lighting conditions. To address the problem, this paper presents an Event-based SAI (E-SAI) method by relying on the asynchronous events with extremely low latency and high dynamic range acquired by an event camera. Specifically, the collected events are first refocused by a Refocus-Net module to align in-focus events while scattering out off-focus ones. Following that, a hybrid network composed of spiking neural networks (SNNs) and convolutional neural networks (CNNs) is proposed to encode the spatio-temporal information from the refocused events and reconstruct a visual image of the occluded targets. Extensive experiments demonstrate that our proposed E-SAI method can achieve remarkable performance in dealing with very dense occlusions and extreme lighting conditions and produce high-quality images from pure events. Codes and datasets are available at https://dvs-whu.cn/projects/esai/.