GenState-AI: State-Aware Dataset for Text-to-Video Retrieval on AI-Generated Videos

TL;DR

GenState-AI introduces a novel benchmark for text-to-video retrieval emphasizing controlled state transitions and end-state grounding, revealing challenges in temporal and semantic discrimination for current models.

Contribution

The paper presents GenState-AI, a new dataset and evaluation framework focused on state-aware retrieval with fine-grained temporal and semantic distinctions, and provides diagnostic analyses of model failures.

Findings

Models often confuse temporal hard negatives with main videos.

Models tend to over-prefer temporally plausible but incorrect end-states.

Semantic substitutions are less influential in model confusion patterns.

Abstract

Existing text-to-video retrieval benchmarks are dominated by real-world footage where much of the semantics can be inferred from a single frame, leaving temporal reasoning and explicit end-state grounding under-evaluated. We introduce GenState-AI, an AI-generated benchmark centered on controlled state transitions, where each query is paired with a main video, a temporal hard negative that differs only in the decisive end-state, and a semantic hard negative with content substitution, enabling fine-grained diagnosis of temporal vs. semantic confusions beyond appearance matching. Using Wan2.2-TI2V-5B, we generate short clips whose meaning depends on precise changes in position, quantity, and object relations, providing controllable evaluation conditions for state-aware retrieval. We evaluate two representative MLLM-based baselines, and observe consistent and interpretable failure patterns: both frequently confuse the main video with the temporal hard negative and over-prefer temporally plausible but end-state-incorrect clips, indicating insufficient grounding to decisive end-state evidence, while being comparatively less sensitive to semantic substitutions. We further introduce triplet-based diagnostic analyses, including relative-order statistics and breakdowns across transition categories, to make temporal vs. semantic failure sources explicit. GenState-AI provides a focused testbed for state-aware, temporally and semantically sensitive text-to-video retrieval, and will be released on huggingface.co.