TSPO: Temporal Sampling Policy Optimization for Long-form Video Language Understanding

TL;DR

TSPO introduces a reinforcement learning-based approach to optimize frame sampling in long-form video language models, significantly improving their understanding and performance on various benchmarks.

Contribution

The paper presents a novel trainable, event-aware temporal sampling policy optimized via reinforcement learning for enhanced long-video understanding in multimodal language models.

Findings

Achieves state-of-the-art results on multiple benchmarks.

Demonstrates transferability across different Video-MLLMs.

Balances temporal understanding and key segment localization effectively.

Abstract

Multimodal Large Language Models (MLLMs) have demonstrated significant progress in vision-language tasks, yet they still face challenges when processing long-duration video inputs. The limitation arises from MLLMs' context limit and training costs, necessitating sparse frame sampling before feeding videos into MLLMs. However, building a trainable sampling method remains challenging due to the unsupervised and non-differentiable nature of sparse frame sampling in Video-MLLMs. To address these problems, we propose Temporal Sampling Policy Optimization (TSPO), advancing MLLMs' long-form video-language understanding via reinforcement learning. Specifically, we first propose a trainable event-aware temporal agent, which captures event-query correlation for performing probabilistic keyframe selection. Then, we propose the TSPO reinforcement learning paradigm, which models keyframe selection and language generation as a joint decision-making process, enabling end-to-end group relative optimization for the temporal sampling policy. Furthermore, we propose a dual-style long video training data construction pipeline, balancing comprehensive temporal understanding and key segment localization. Finally, we incorporate rule-based answering accuracy and temporal locating reward mechanisms to optimize the temporal sampling policy. Comprehensive experiments show that our TSPO achieves state-of-the-art performance across multiple long video understanding benchmarks, and shows transferable ability across different cutting-edge Video-MLLMs. Our code is available at https://github.com/Hui-design/TSPO