Need for Speed: Zero-Shot Depth Completion with Single-Step Diffusion

TL;DR

Marigold-SSD is a fast, zero-shot depth completion framework that uses diffusion priors and shifts computation to finetuning, enabling efficient 3D perception without test-time optimization.

Contribution

It introduces a single-step, late-fusion depth completion method that reduces inference time and training cost while maintaining strong cross-domain generalization.

Findings

Achieves faster inference with only 4.5 GPU days of training.

Demonstrates strong zero-shot performance across multiple benchmarks.

Narrows the efficiency gap between diffusion-based and discriminative models.

Abstract

We introduce Marigold-SSD, a single-step, late-fusion depth completion framework that leverages strong diffusion priors while eliminating the costly test-time optimization typically associated with diffusion-based methods. By shifting computational burden from inference to finetuning, our approach enables efficient and robust 3D perception under real-world latency constraints. Marigold-SSD achieves significantly faster inference with a training cost of only 4.5 GPU days. We evaluate our method across four indoor and two outdoor benchmarks, demonstrating strong cross-domain generalization and zero-shot performance compared to existing depth completion approaches. Our approach significantly narrows the efficiency gap between diffusion-based and discriminative models. Finally, we challenge common evaluation protocols by analyzing performance under varying input sparsity levels. Page: https://dtu-pas.github.io/marigold-ssd/