BrokenVideos: A Benchmark Dataset for Fine-Grained Artifact Localization in AI-Generated Videos

TL;DR

BrokenVideos is a new benchmark dataset with pixel-level annotations for evaluating artifact localization in AI-generated videos, addressing a key gap in the field and enabling improved detection methods.

Contribution

We introduce BrokenVideos, a comprehensive dataset with detailed annotations for fine-grained artifact localization in AI-generated videos, facilitating research and development of better detection models.

Findings

Training models on BrokenVideos improves artifact localization accuracy.

Fine-grained annotations enable better evaluation of generative video quality.

Benchmark establishes a foundation for future research in artifact detection.

Abstract

Recent advances in deep generative models have led to significant progress in video generation, yet the fidelity of AI-generated videos remains limited. Synthesized content often exhibits visual artifacts such as temporally inconsistent motion, physically implausible trajectories, unnatural object deformations, and local blurring that undermine realism and user trust. Accurate detection and spatial localization of these artifacts are crucial for both automated quality control and for guiding the development of improved generative models. However, the research community currently lacks a comprehensive benchmark specifically designed for artifact localization in AI generated videos. Existing datasets either restrict themselves to video or frame level detection or lack the fine-grained spatial annotations necessary for evaluating localization methods. To address this gap, we introduce BrokenVideos, a benchmark dataset of 3,254 AI-generated videos with meticulously annotated, pixel-level masks highlighting regions of visual corruption. Each annotation is validated through detailed human inspection to ensure high quality ground truth. Our experiments show that training state of the art artifact detection models and multi modal large language models (MLLMs) on BrokenVideos significantly improves their ability to localize corrupted regions. Through extensive evaluation, we demonstrate that BrokenVideos establishes a critical foundation for benchmarking and advancing research on artifact localization in generative video models. The dataset is available at: https://broken-video-detection-datetsets.github.io/Broken-Video-Detection-Datasets.github.io/.