Video Finetuning Improves Reasoning Between Frames

TL;DR

This paper introduces Visual Chain-of-Thought (vCoT), a reasoning process that enhances multimodal LLMs' understanding of video frame transitions, improving long-form video question answering and relational reasoning.

Contribution

It proposes vCoT for explicit temporal reasoning in videos and systematically compares image-only and video-finetuned models, revealing insights into their reasoning capabilities.

Findings

vCoT improves long-form video question answering performance.

Video-finetuned models implicitly capture frame transitions.

Video models transfer temporal reasoning to static visual tasks.

Abstract

Multimodal large language models (LLMs) have made rapid progress in visual understanding, yet their extension from images to videos often reduces to a naive concatenation of frame tokens. In this work, we investigate what video finetuning brings to multimodal LLMs. We propose Visual Chain-of-Thought (vCoT), an explicit reasoning process that generates transitional event descriptions between consecutive frames. Using vCoT, we systematically compare image-only LVLMs with their video-finetuned counterparts, both with and without access to these transitional cues. Our experiments show that vCoT significantly improves the performance of image-only models on long-form video question answering, while yielding only marginal gains for video-finetuned models. This suggests that the latter already capture frame-to-frame transitions implicitly. Moreover, we find that video models transfer this temporal reasoning ability to purely static settings, outperforming image models' baselines on relational visual reasoning tasks.