Learning Temporally Consistent Video Depth from Video Diffusion Priors

TL;DR

This paper introduces ChronoDepth, a novel method for streamed video depth estimation that ensures cross-frame consistency by reformulating depth prediction as a conditional generation problem with a context-aware training strategy.

Contribution

It proposes a new framework that leverages contextual information across frames and clips to improve temporal consistency in video depth estimation.

Findings

Outperforms existing methods in maintaining temporal consistency.

Effective in arbitrarily long videos with cross-clip context.

Validated through extensive experiments.

Abstract

This work addresses the challenge of streamed video depth estimation, which expects not only per-frame accuracy but, more importantly, cross-frame consistency. We argue that sharing contextual information between frames or clips is pivotal in fostering temporal consistency. Therefore, we reformulate depth prediction into a conditional generation problem to provide contextual information within a clip and across clips. Specifically, we propose a consistent context-aware training and inference strategy for arbitrarily long videos to provide cross-clip context. We sample independent noise levels for each frame within a clip during training while using a sliding window strategy and initializing overlapping frames with previously predicted frames without adding noise. Moreover, we design an effective training strategy to provide context within a clip. Extensive experimental results validate our design choices and demonstrate the superiority of our approach, dubbed ChronoDepth. Project page: https://xdimlab.github.io/ChronoDepth/.