TwiFF (Think With Future Frames): A Large-Scale Dataset for Dynamic Visual Reasoning

TL;DR

TwiFF introduces a large-scale, temporally grounded video reasoning dataset and a novel model that enhances dynamic visual reasoning by integrating future frame prediction with question answering, significantly outperforming existing methods.

Contribution

The paper presents TwiFF-2.7M, a large-scale dataset for dynamic visual reasoning, and TwiFF, a model that combines video generation and comprehension for improved temporal reasoning.

Findings

TwiFF outperforms existing VCoT methods on dynamic reasoning tasks.

The dataset enables better training and evaluation of temporal reasoning models.

TwiFF model effectively generates temporally coherent future frames for reasoning.

Abstract

Visual Chain-of-Thought (VCoT) has emerged as a promising paradigm for enhancing multimodal reasoning by integrating visual perception into intermediate reasoning steps. However, existing VCoT approaches are largely confined to static scenarios and struggle to capture the temporal dynamics essential for tasks such as instruction, prediction, and camera motion. To bridge this gap, we propose TwiFF-2.7M, the first large-scale, temporally grounded VCoT dataset derived from $2.7$ million video clips, explicitly designed for dynamic visual question and answer. Accompanying this, we introduce TwiFF-Bench, a high-quality evaluation benchmark of $1,078$ samples that assesses both the plausibility of reasoning trajectories and the correctness of final answers in open-ended dynamic settings. Building on these foundations, we propose the TwiFF model, a unified modal that synergistically leverages pre-trained video generation and image comprehension capabilities to produce temporally coherent visual reasoning cues-iteratively generating future action frames and textual reasoning. Extensive experiments demonstrate that TwiFF significantly outperforms existing VCoT methods and Textual Chain-of-Thought baselines on dynamic reasoning tasks, which fully validates the effectiveness for visual question answering in dynamic scenarios. Our code and data is available at https://github.com/LiuJunhua02/TwiFF.