StableDPT: Temporal Stable Monocular Video Depth Estimation

TL;DR

StableDPT is a novel method that enhances monocular video depth estimation by integrating a temporal module into existing models, significantly improving stability and efficiency across benchmarks.

Contribution

It introduces a trainable temporal layer with cross-attention in a Vision Transformer-based architecture, enabling stable, global context-aware depth estimation for videos.

Findings

Improves temporal stability and consistency in depth predictions.

Achieves state-of-the-art performance on benchmark datasets.

Processes videos twice as fast as previous methods.

Abstract

Applying single image Monocular Depth Estimation (MDE) models to video sequences introduces significant temporal instability and flickering artifacts. We propose a novel approach that adapts any state-of-the-art image-based (depth) estimation model for video processing by integrating a new temporal module - trainable on a single GPU in a few days. Our architecture StableDPT builds upon an off-the-shelf Vision Transformer (ViT) encoder and enhances the Dense Prediction Transformer (DPT) head. The core of our contribution lies in the temporal layers within the head, which use an efficient cross-attention mechanism to integrate information from keyframes sampled across the entire video sequence. This allows the model to capture global context and inter-frame relationships leading to more accurate and temporally stable depth predictions. Furthermore, we propose a novel inference strategy for processing videos of arbitrary length avoiding the scale misalignment and redundant computations associated with overlapping windows used in other methods. Evaluations on multiple benchmark datasets demonstrate improved temporal consistency, competitive state-of-the-art performance and on top 2x faster processing in real-world scenarios.