A Study of Finetuning Video Transformers for Multi-view Geometry Tasks

TL;DR

This paper demonstrates that general-purpose video foundation models can be effectively fine-tuned for multi-view geometry tasks like optical flow, achieving state-of-the-art results with minimal architectural modifications.

Contribution

It shows that pretrained video transformers can be adapted to geometric tasks using simple linear decoders and iterative refinement, without task-specific pretraining or complex architectures.

Findings

Achieved top cross-dataset generalization for optical flow.

Set new records on online benchmarks for optical flow.

Strong performance in 3D depth estimation and stereo matching.

Abstract

This paper presents an investigation of vision transformer learning for multi-view geometry tasks, such as optical flow estimation, by fine-tuning video foundation models. Unlike previous methods that involve custom architectural designs and task-specific pretraining, our research finds that general-purpose models pretrained on videos can be readily transferred to multi-view problems with minimal adaptation. The core insight is that general-purpose attention between patches learns temporal and spatial information for geometric reasoning. We demonstrate that appending a linear decoder to the Transformer backbone produces satisfactory results, and iterative refinement can further elevate performance to stateof-the-art levels. This conceptually simple approach achieves top cross-dataset generalization results for optical flow estimation with end-point error (EPE) of 0.69, 1.78, and 3.15 on the Sintel clean, Sintel final, and KITTI datasets, respectively. Our method additionally establishes a new record on the online test benchmark with EPE values of 0.79, 1.88, and F1 value of 3.79. Applications to 3D depth estimation and stereo matching also show strong performance, illustrating the versatility of video-pretrained models in addressing geometric vision tasks.