ViTCoT: Video-Text Interleaved Chain-of-Thought for Boosting Video Understanding in Large Language Models

TL;DR

This paper introduces ViTCoT, a novel video reasoning paradigm that interleaves visual and textual information to improve understanding in large language models, inspired by human cognitive processes.

Contribution

The paper proposes ViTCoT, a new interleaved video-text reasoning framework and a benchmark dataset, enhancing video understanding in large language models beyond text-only methods.

Findings

ViTCoT outperforms traditional text-only CoT in video reasoning tasks.

It activates more neuron values in multimodal large language models.

The approach demonstrates significant performance improvements in experiments.

Abstract

Video understanding plays a vital role in bridging low-level visual signals with high-level cognitive reasoning, and is fundamental to applications such as autonomous driving, embodied AI, and the broader pursuit of AGI. The rapid development of large language models (LLMs), particularly those utilizing Chain-of-Thought (CoT) technology, has significantly advanced video reasoning capabilities. However, current approaches primarily depend on textual information for reasoning, overlooking the visual modality in the actual video reasoning process. In contrast, humans naturally re-examine visual content while reasoning. Motivated by this, we introduce a novel video reasoning paradigm: Video-Text Interleaved CoT (ViTCoT), which facilitates more intuitive and cognitively aligned reasoning. To the end, first, we construct the Video-Text Interleaved Benchmark (ViTIB), which is created using MLLMs for key-video selection and manually verified. Furthermore, we extensively explore the potential of the ViTCoT paradigm in the video understanding field. Extensive experiments demonstrate that ViTCoT significantly enhances performance compared to the traditional text-only CoT paradigm and effectively activates more neuron values in MLLMs.