ViDiC: Video Difference Captioning

TL;DR

ViDiC introduces a new task and dataset for evaluating how well multimodal models can describe and compare dynamic video scenes, emphasizing motion, event evolution, and editing consistency.

Contribution

The paper presents the ViDiC task and ViDiC-1K dataset, along with a dual-checklist evaluation framework, to advance video difference captioning and comparative reasoning in multimodal models.

Findings

Significant performance gap in existing models' ability to describe video differences

ViDiC-1K provides a challenging benchmark for video understanding

Dual-checklist framework effectively measures similarity and difference accuracy

Abstract

Understanding visual differences between dynamic scenes requires the comparative perception of compositional, spatial, and temporal changes--a capability that remains underexplored in existing vision-language systems. While prior work on Image Difference Captioning (IDC) has enabled models to describe semantic changes between static images, these approaches fail to capture motion continuity, event evolution, or editing consistency over time. We introduce the ViDiC (Video Difference Captioning) task and its corresponding ViDiC-1K dataset, designed to evaluate the ability of Multimodal Large Language Models (MLLMs) to provide fine-grained descriptions of similarities and differences between video pairs. ViDiC-1K comprises 1,000 curated video pairs annotated with over 4,000 comparative checklist items, covering seven categories: subject, style, background, cinematography, motion, location, and playback techniques. To ensure reliable evaluation, we propose a dual-checklist framework that measures the accuracy of similarity and difference separately, based on the LLM-as-a-Judge protocol. Experiments on nineteen representative multimodal models reveal a significant performance gap in their comparative description and difference perception abilities. We hope ViDiC-1K can be a challenging benchmark that lays a solid foundation for advancing video understanding, edit awareness, and comparative reasoning in multimodal intelligence.