Temporal Gains, Spatial Costs: Revisiting Video Fine-Tuning in Multimodal Large Language Models

TL;DR

This paper investigates how video-based fine-tuning affects multimodal large language models, revealing a trade-off between improved video understanding and degraded static image performance, and proposing an adaptive frame sampling strategy.

Contribution

It systematically analyzes the impact of Video-SFT on visual capabilities, highlighting the spatial-temporal trade-off and introducing an instruction-aware hybrid sampling method.

Findings

Video-SFT improves video performance but can degrade static image benchmarks.

Increasing sampled frames boosts video understanding but not static image performance.

Adaptive frame allocation partially mitigates the image-video trade-off.

Abstract

Multimodal large language models (MLLMs) are typically trained in multiple stages, with video-based supervised fine-tuning (Video-SFT) serving as a key step for improving visual understanding. Yet its effect on the fine-grained evolution of visual capabilities, particularly the balance between spatial and temporal understanding, remains poorly understood. In this paper, we systematically study how Video-SFT reshapes visual capabilities in MLLMs. Across architectures, parameter scales, and frame sampling settings, we observe a consistent pattern: Video-SFT reliably improves video performance, but often yields limited gains or even degradation on static image benchmarks. We further show that this trade-off is closely tied to temporal budget: increasing the number of sampled frames generally improves video performance, but does not reliably improve static image performance. Motivated by this finding, we study an instruction-aware Hybrid-Frame strategy that adaptively allocates frame counts and partially mitigates the image-video trade-off. Our results indicate that Video-SFT is not a free lunch for MLLMs, and preserving spatial understanding remains a central challenge in joint image-video training.