Event-Driven Dynamic Scene Depth Completion

TL;DR

EventDC is a novel event-driven depth completion framework that leverages high-temporal-resolution event data to improve depth estimation in dynamic scenes, outperforming existing methods.

Contribution

We introduce EventDC, the first framework utilizing event cameras for depth completion, with novel modules for motion-aware feature alignment and local depth refinement.

Findings

EventDC outperforms existing depth completion methods in dynamic scenes.

We establish the first benchmark for event-based depth completion with real and synthetic datasets.

Experimental results show significant improvements in depth accuracy and alignment.

Abstract

Depth completion in dynamic scenes poses significant challenges due to rapid ego-motion and object motion, which can severely degrade the quality of input modalities such as RGB images and LiDAR measurements. Conventional RGB-D sensors often struggle to align precisely and capture reliable depth under such conditions. In contrast, event cameras with their high temporal resolution and sensitivity to motion at the pixel level provide complementary cues that are %particularly beneficial in dynamic environments.To this end, we propose EventDC, the first event-driven depth completion framework. It consists of two key components: Event-Modulated Alignment (EMA) and Local Depth Filtering (LDF). Both modules adaptively learn the two fundamental components of convolution operations: offsets and weights conditioned on motion-sensitive event streams. In the encoder, EMA leverages events to modulate the sampling positions of RGB-D features to achieve pixel redistribution for improved alignment and fusion. In the decoder, LDF refines depth estimations around moving objects by learning motion-aware masks from events. Additionally, EventDC incorporates two loss terms to further benefit global alignment and enhance local depth recovery. Moreover, we establish the first benchmark for event-based depth completion comprising one real-world and two synthetic datasets to facilitate future research. Extensive experiments on this benchmark demonstrate the superiority of our EventDC.