Scaling RL to Long Videos

TL;DR

This paper presents a comprehensive framework for scaling reinforcement learning to long videos in vision-language models, including a large dataset, a novel training pipeline, and an efficient training infrastructure, achieving state-of-the-art results and supporting diverse modalities.

Contribution

The paper introduces LongVideo-Reason dataset, a two-stage training pipeline with chain-of-thought fine-tuning and RL, and a new training infrastructure MR-SP for long video reasoning in VLMs.

Findings

Achieved 65.1% accuracy on VideoMME without subtitles.

Supported processing of up to 8,192 video frames per video.

MR-SP system speeds up long video RL training by 2.1x.

Abstract

We introduce a full-stack framework that scales up reasoning in vision-language models (VLMs) to long videos, leveraging reinforcement learning. We address the unique challenges of long video reasoning by integrating three critical components: (1) a large-scale dataset, LongVideo-Reason, comprising 104K long video QA pairs with high-quality reasoning annotations across diverse domains such as sports, games, and vlogs; (2) a two-stage training pipeline that extends VLMs with chain-of-thought supervised fine-tuning (CoT-SFT) and reinforcement learning (RL); and (3) a training infrastructure for long video RL, named Multi-modal Reinforcement Sequence Parallelism (MR-SP), which incorporates sequence parallelism and a vLLM-based engine tailored for long video, using cached video embeddings for efficient rollout and prefilling. In our experiments, LongVILA-R1-7B achieves strong performance on video benchmarks, reaching 65.1% and 71.1% accuracy on VideoMME without and with subtitles, respectively, and consistently outperforming LongVILA-7B across multiple benchmarks. Moreover, LongVILA-R1-7B supports processing up to 8,192 video frames per video, and configurable FPS settings. Notably, our MR-SP system achieves up to 2.1x speedup on long video RL training. In addition, we release our training system for public availability that supports RL training on various modalities (video, text, and audio), various models (VILA and Qwen series), and even image and video generation models. On a single A100 node (8 GPUs), it supports RL training on hour-long videos (e.g., 3,600 frames).