Are Synthetic Videos Useful? A Benchmark for Retrieval-Centric Evaluation of Synthetic Videos
Zecheng Zhao, Selena Song, Tong Chen, Zhi Chen, Shazia Sadiq, Yadan Luo

TL;DR
This paper introduces SynTVA, a benchmark and dataset for evaluating the usefulness of synthetic videos in text-to-video retrieval tasks, highlighting the importance of semantic alignment over traditional quality metrics.
Contribution
The work presents a new dataset, benchmark, and evaluation framework for assessing synthetic videos' utility in retrieval tasks, including an Auto-Evaluator for estimating alignment quality.
Findings
Semantic alignment correlates with retrieval performance.
Synthetic videos can enhance dataset utility for TVR.
Auto-Evaluator predicts alignment quality from existing metrics.
Abstract
Text-to-video (T2V) synthesis has advanced rapidly, yet current evaluation metrics primarily capture visual quality and temporal consistency, offering limited insight into how synthetic videos perform in downstream tasks such as text-to-video retrieval (TVR). In this work, we introduce SynTVA, a new dataset and benchmark designed to evaluate the utility of synthetic videos for building retrieval models. Based on 800 diverse user queries derived from MSRVTT training split, we generate synthetic videos using state-of-the-art T2V models and annotate each video-text pair along four key semantic alignment dimensions: Object \& Scene, Action, Attribute, and Prompt Fidelity. Our evaluation framework correlates general video quality assessment (VQA) metrics with these alignment scores, and examines their predictive power for downstream TVR performance. To explore pathways of scaling up, we…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
