ViPE: Video Pose Engine for 3D Geometric Perception

TL;DR

ViPE is a versatile video processing engine that accurately estimates camera parameters and dense depth maps from unconstrained videos, enabling large-scale annotation for spatial AI applications.

Contribution

Introduces ViPE, a robust, efficient tool for estimating camera pose and depth from diverse videos, and provides a large annotated dataset to advance spatial AI research.

Findings

Outperforms existing pose estimation baselines by 18%/50% on TUM/KITTI.

Runs at 3-5 FPS on a single GPU.

Annotated approximately 96 million frames across various video types.

Abstract

Accurate 3D geometric perception is an important prerequisite for a wide range of spatial AI systems. While state-of-the-art methods depend on large-scale training data, acquiring consistent and precise 3D annotations from in-the-wild videos remains a key challenge. In this work, we introduce ViPE, a handy and versatile video processing engine designed to bridge this gap. ViPE efficiently estimates camera intrinsics, camera motion, and dense, near-metric depth maps from unconstrained raw videos. It is robust to diverse scenarios, including dynamic selfie videos, cinematic shots, or dashcams, and supports various camera models such as pinhole, wide-angle, and 360{\deg} panoramas. We have benchmarked ViPE on multiple benchmarks. Notably, it outperforms existing uncalibrated pose estimation baselines by 18%/50% on TUM/KITTI sequences, and runs at 3-5FPS on a single GPU for standard input resolutions. We use ViPE to annotate a large-scale collection of videos. This collection includes around 100K real-world internet videos, 1M high-quality AI-generated videos, and 2K panoramic videos, totaling approximately 96M frames -- all annotated with accurate camera poses and dense depth maps. We open-source ViPE and the annotated dataset with the hope of accelerating the development of spatial AI systems.