SceneScribe-1M: A Large-Scale Video Dataset with Comprehensive Geometric and Semantic Annotations

TL;DR

SceneScribe-1M is a large-scale, multi-modal video dataset with detailed annotations supporting 3D understanding and video synthesis, aiming to advance research in both domains.

Contribution

It introduces a comprehensive dataset with rich annotations for 3D perception and video generation, filling a gap in existing resources.

Findings

Established benchmarks for depth estimation, scene reconstruction, and point tracking.

Demonstrated the dataset's versatility across multiple downstream tasks.

Open-sourced the dataset to foster further research.

Abstract

The convergence of 3D geometric perception and video synthesis has created an unprecedented demand for large-scale video data that is rich in both semantic and spatio-temporal information. While existing datasets have advanced either 3D understanding or video generation, a significant gap remains in providing a unified resource that supports both domains at scale. To bridge this chasm, we introduce SceneScribe-1M, a new large-scale, multi-modal video dataset. It comprises one million in-the-wild videos, each meticulously annotated with detailed textual descriptions, precise camera parameters, dense depth maps, and consistent 3D point tracks. We demonstrate the versatility and value of SceneScribe-1M by establishing benchmarks across a wide array of downstream tasks, including monocular depth estimation, scene reconstruction, and dynamic point tracking, as well as generative tasks such as text-to-video synthesis, with or without camera control. By open-sourcing SceneScribe-1M, we aim to provide a comprehensive benchmark and a catalyst for research, fostering the development of models that can both perceive the dynamic 3D world and generate controllable, realistic video content.