Video-RTS: Rethinking Reinforcement Learning and Test-Time Scaling for Efficient and Enhanced Video Reasoning

TL;DR

Video-RTS introduces a data-efficient reinforcement learning approach combined with adaptive test-time scaling for improved video reasoning, reducing reliance on extensive fine-tuning and annotations while achieving superior accuracy on benchmarks.

Contribution

It proposes a novel combination of pure RL training and a sparse-to-dense TTS strategy, significantly enhancing data efficiency and inference performance in video reasoning tasks.

Findings

Surpasses existing models by 2.4% accuracy with only 3.6% of training data

Achieves a 4.2% improvement on Video-Holmes benchmark

Eliminates the need for large-scale supervised fine-tuning

Abstract

Despite advances in reinforcement learning (RL)-based video reasoning with large language models (LLMs), data collection and fine-tuning remain significant challenges. These methods often rely on large-scale supervised fine-tuning (SFT) with extensive video data and long Chain-of-Thought (CoT) annotations, making them costly and hard to scale. To address this, we present Video-RTS, a new approach to improve video reasoning capability with drastically improved data efficiency by combining data-efficient RL with a video-adaptive test-time scaling (TTS) strategy. Building on observations about the data scaling, we skip the resource-intensive SFT step and employ efficient pure-RL training with output-based rewards, requiring no additional annotations or extensive fine-tuning. Furthermore, to utilize computational resources more efficiently, we introduce a sparse-to-dense video TTS strategy that improves inference by iteratively adding frames based on output consistency. We validate our approach on multiple video reasoning benchmarks, showing that Video-RTS surpasses existing video reasoning models by 2.4% in accuracy using only 3.6% training samples. Specifically, Video-RTS achieves a 4.2% improvement on Video-Holmes, a recent and challenging video reasoning benchmark. Notably, our pure RL training and adaptive video TTS offer complementary strengths, enabling Video-RTS's strong reasoning performance.