Temporal Chain of Thought: Long-Video Understanding by Thinking in Frames

TL;DR

The paper introduces Temporal Chain of Thought, an inference strategy that improves long-video understanding by selectively extracting relevant frames for question-answering, achieving state-of-the-art results especially on very long videos.

Contribution

It proposes a novel inference method that dynamically selects relevant frames, enhancing long-video comprehension beyond fixed context window limitations.

Findings

State-of-the-art results on 4 video QA datasets

Significant improvement on videos longer than 1 hour

Effective frame selection enhances accuracy with limited context

Abstract

Despite recent advances in Vision-Language Models (VLMs), long-video understanding remains a challenging problem. Although state-of-the-art long-context VLMs can process around 1000 input frames, they still struggle to effectively leverage this sequence length, and succumb to irrelevant distractors within the context window. We present Temporal Chain of Thought, an inference strategy for video question-answering that curates the model's input context. We use the VLM itself to iteratively identify and extract the most relevant frames from the video, which are then used for answering. We demonstrate how leveraging more computation at inference-time to select the most relevant context leads to improvements in accuracy, in agreement with recent work on inference-time scaling of LLMs. Moreover, we achieve state-of-the-art results on 4 diverse video question-answering datasets, showing consistent improvements with 3 different VLMs. In particular, our method shines on longer videos which would not otherwise fit within the model's context window: On longer videos of more than 1 hour on LVBench, our approach using a context window of 32K outperforms the same VLM using standard inference with a 700K context window by 2.8 points.