VDCook:DIY video data cook your MLLMs

TL;DR

VDCook is a configurable, self-evolving video data platform that automates data construction, annotation, and continuous updating for machine learning applications, lowering barriers for specialized dataset creation.

Contribution

It introduces a novel, automated system for dynamic video dataset creation and expansion, integrating query-based data retrieval, synthesis, and multi-dimensional metadata annotation.

Findings

Enables continuous dataset updates through automated ingestion.

Supports domain-specific video data generation with provenance and metadata.

Facilitates community-driven dataset expansion and governance.

Abstract

We introduce VDCook: a self-evolving video data operating system, a configurable video data construction platform for researchers and vertical domain teams. Users initiate data requests via natural language queries and adjustable parameters (scale, retrieval-synthesis ratio, quality threshold). The system automatically performs query optimization, concurrently running real video retrieval and controlled synthesis modules. It ultimately generates in-domain data packages with complete provenance and metadata, along with reproducible Notebooks. Unlike traditional static, one-time-built datasets, VDCook enables continuous updates and domain expansion through its automated data ingestion mechanism based on MCP (Model Context Protocol)\cite{mcp2024anthropic}, transforming datasets into dynamically evolving open ecosystems. The system also provides multi-dimensional metadata annotation (scene segmentation, motion scoring, OCR ratio, automatic captioning, etc.), laying the foundation for flexible subsequent data `cooking' and indexing\cite{vlogger}. This platform aims to significantly lower the barrier to constructing specialized video training datasets through infrastructure-level solutions, while supporting community contributions and a governance-enabled data expansion paradigm. \textbf{Project demo:} https://screenapp.io/app/v/WP0SvffgsH