Reinforcement Learning Tuning for VideoLLMs: Reward Design and Data Efficiency

TL;DR

This paper enhances multimodal large language models for video understanding by introducing a reward-based reinforcement learning tuning method that improves reasoning capabilities with less data, emphasizing reward design and data selection.

Contribution

It proposes a dual-reward reinforcement learning framework based on GRPO for video reasoning, with a variance-aware data selection strategy, improving performance over supervised fine-tuning.

Findings

Outperforms supervised fine-tuning and existing RLT methods.

Achieves superior results across eight video understanding tasks.

Requires significantly less training data.

Abstract

Understanding real-world videos with complex semantics and long temporal dependencies remains a fundamental challenge in computer vision. Recent progress in multimodal large language models (MLLMs) has demonstrated strong capabilities in vision-language tasks, while reinforcement learning tuning (RLT) has further improved their reasoning abilities. In this work, we explore RLT as a post-training strategy to enhance the video-specific reasoning capabilities of MLLMs. Built upon the Group Relative Policy Optimization (GRPO) framework, we propose a dual-reward formulation that supervises both semantic and temporal reasoning through discrete and continuous reward signals. To facilitate effective preference-based optimization, we introduce a variance-aware data selection strategy based on repeated inference to identify samples that provide informative learning signals. We evaluate our approach across eight representative video understanding tasks, including VideoQA, Temporal Video Grounding, and Grounded VideoQA. Our method consistently outperforms supervised fine-tuning and existing RLT baselines, achieving superior performance with significantly less training data. These results underscore the importance of reward design and data selection in advancing reasoning-centric video understanding with MLLMs. Notably, The initial code release (two months ago) has now been expanded with updates, including optimized reward mechanisms and additional datasets. The latest version is available at https://github.com/appletea233/Temporal-R1 .