SAGE: Training Smart Any-Horizon Agents for Long Video Reasoning with Reinforcement Learning

TL;DR

SAGE is a reinforcement learning-based system enabling flexible, multi-turn reasoning over long videos, significantly improving performance on long-video reasoning tasks.

Contribution

The paper introduces SAGE, a novel agent system with synthetic training data and RL post-training, for efficient any-horizon video reasoning.

Findings

Up to 6.1% improvement on open-ended reasoning tasks.

8.2% boost on videos longer than 10 minutes.

Effective RL recipe enhances reasoning ability.

Abstract

As humans, we are natural any-horizon reasoners, i.e., we can decide whether to iteratively skim long videos or watch short ones in full when necessary for a given task. With this in mind, one would expect video reasoning models to reason flexibly across different durations. However, SOTA models are still trained to predict answers in a single turn while processing a large number of frames, akin to watching an entire long video, requiring significant resources. This raises the question: Is it possible to develop performant any-horizon video reasoning systems? Inspired by human behavior, we first propose SAGE, an agent system that performs multi-turn reasoning on long videos while handling simpler problems in a single turn. Secondly, we introduce an easy synthetic data generation pipeline using Gemini-2.5-Flash to train the orchestrator, SAGE-MM, which lies at the core of SAGE. We further propose an effective RL post-training recipe essential for instilling any-horizon reasoning ability in SAGE-MM. Thirdly, we curate SAGE-Bench with an average duration of greater than 700 seconds for evaluating video reasoning ability in real-world entertainment use cases. Lastly, we empirically validate the effectiveness of our system, data, and RL recipe, observing notable improvements of up to 6.1% on open-ended video reasoning tasks, as well as an impressive 8.2% improvement on videos longer than 10 minutes.