Unsupervised Spike Depth Estimation via Cross-modality Cross-domain Knowledge Transfer

TL;DR

This paper introduces a novel unsupervised framework for spike depth estimation that leverages open-source RGB data through cross-modality and cross-domain knowledge transfer, achieving state-of-the-art results.

Contribution

It proposes the BiCross framework with CFKD and SCTS mechanisms to enable effective unsupervised spike depth estimation from RGB data despite modality and domain differences.

Findings

Achieves state-of-the-art performance on multiple scenarios.

Effectively transfers knowledge from RGB to spike data.

Reduces error accumulation with self-correcting mechanisms.

Abstract

Neuromorphic spike data, an upcoming modality with high temporal resolution, has shown promising potential in autonomous driving by mitigating the challenges posed by high-velocity motion blur. However, training the spike depth estimation network holds significant challenges in two aspects: sparse spatial information for pixel-wise tasks and difficulties in achieving paired depth labels for temporally intensive spike streams. Therefore, we introduce open-source RGB data to support spike depth estimation, leveraging its annotations and spatial information. The inherent differences in modalities and data distribution make it challenging to directly apply transfer learning from open-source RGB to target spike data. To this end, we propose a cross-modality cross-domain (BiCross) framework to realize unsupervised spike depth estimation by introducing simulated mediate source spike data. Specifically, we design a Coarse-to-Fine Knowledge Distillation (CFKD) approach to facilitate comprehensive cross-modality knowledge transfer while preserving the unique strengths of both modalities, utilizing a spike-oriented uncertainty scheme. Then, we propose a Self-Correcting Teacher-Student (SCTS) mechanism to screen out reliable pixel-wise pseudo labels and ease the domain shift of the student model, which avoids error accumulation in target spike data. To verify the effectiveness of BiCross, we conduct extensive experiments on four scenarios, including Synthetic to Real, Extreme Weather, Scene Changing, and Real Spike. Our method achieves state-of-the-art (SOTA) performances, compared with RGB-oriented unsupervised depth estimation methods. Code and dataset: https://github.com/Theia-4869/BiCross