CounterVid: Counterfactual Video Generation for Mitigating Action and Temporal Hallucinations in Video-Language Models

TL;DR

CounterVid introduces a scalable framework for generating counterfactual videos to reduce hallucinations in video-language models, improving temporal reasoning and action recognition by synthesizing challenging negative examples.

Contribution

The paper presents a novel counterfactual video generation method and a unified preference optimization approach, enhancing model robustness and temporal understanding.

Findings

Improved temporal ordering accuracy in VLMs.

CounterVid dataset with 26k preference pairs for training.

Enhanced transferability to standard hallucination benchmarks.

Abstract

Video-language models (VLMs) achieve strong multimodal understanding but remain prone to hallucinations, especially when reasoning about actions and temporal order. Existing mitigation strategies, such as textual filtering or random video perturbations, often fail to address the root cause: over-reliance on language priors rather than fine-grained visual dynamics. We propose a scalable framework for counterfactual video generation that synthesizes videos differing only in actions or temporal structure while preserving scene context. Our pipeline combines multimodal LLMs for action proposal and editing guidance with diffusion-based image and video models to generate semantic hard negatives at scale. Using this framework, we build CounterVid, a synthetic dataset of ~26k preference pairs targeting action recognition and temporal reasoning. We further introduce MixDPO, a unified Direct Preference Optimization approach that jointly leverages textual and visual preferences. Fine-tuning Qwen2.5-VL with MixDPO yields consistent improvements, notably in temporal ordering, and transfers effectively to standard video hallucination benchmarks. Code and models will be made publicly available.