Self-Supervised Intensity-Event Stereo Matching

TL;DR

This paper introduces a self-supervised stereo matching method that connects event and intensity cameras, enabling high-quality depth estimation without ground truth data, and demonstrates its effectiveness on synthetic and real datasets.

Contribution

It proposes a novel self-supervised multi-modal stereo matching approach that leverages event and intensity images without requiring ground truth disparity.

Findings

Effective disparity estimation on synthetic and real datasets.

Improved robustness with proposed stereo loss functions.

Potential application in downstream tasks like video interpolation.

Abstract

Event cameras are novel bio-inspired vision sensors that output pixel-level intensity changes in microsecond accuracy with a high dynamic range and low power consumption. Despite these advantages, event cameras cannot be directly applied to computational imaging tasks due to the inability to obtain high-quality intensity and events simultaneously. This paper aims to connect a standalone event camera and a modern intensity camera so that the applications can take advantage of both two sensors. We establish this connection through a multi-modal stereo matching task. We first convert events to a reconstructed image and extend the existing stereo networks to this multi-modality condition. We propose a self-supervised method to train the multi-modal stereo network without using ground truth disparity data. The structure loss calculated on image gradients is used to enable self-supervised learning on such multi-modal data. Exploiting the internal stereo constraint between views with different modalities, we introduce general stereo loss functions, including disparity cross-consistency loss and internal disparity loss, leading to improved performance and robustness compared to existing approaches. The experiments demonstrate the effectiveness of the proposed method, especially the proposed general stereo loss functions, on both synthetic and real datasets. At last, we shed light on employing the aligned events and intensity images in downstream tasks, e.g., video interpolation application.