STRIVE: Structured Spatiotemporal Exploration for Reinforcement Learning in Video Question Answering

TL;DR

STRIVE is a structured reinforcement learning framework that enhances video question answering by constructing multiple spatiotemporal variants of videos, enriching reward signals, and promoting stable policy updates.

Contribution

It introduces a novel spatiotemporal variant construction and importance-aware sampling mechanism to improve stability and performance in multimodal reinforcement learning for video reasoning.

Findings

Consistent improvements over baselines on six video reasoning benchmarks.

Enrichment of reward signals through structured visual perturbations.

Enhanced stability and robustness in policy learning.

Abstract

We introduce STRIVE (SpatioTemporal Reinforcement with Importance-aware Variant Exploration), a structured reinforcement learning framework for video question answering. While group-based policy optimization methods have shown promise in large multimodal models, they often suffer from low reward variance when responses exhibit similar correctness, leading to weak or unstable advantage estimates. STRIVE addresses this limitation by constructing multiple spatiotemporal variants of each input video and performing joint normalization across both textual generations and visual variants. By expanding group comparisons beyond linguistic diversity to structured visual perturbations, STRIVE enriches reward signals and promotes more stable and informative policy updates. To ensure exploration remains semantically grounded, we introduce an importance-aware sampling mechanism that prioritizes frames most relevant to the input question while preserving temporal coverage. This design encourages robust reasoning across complementary visual perspectives rather than overfitting to a single spatiotemporal configuration. Experiments on six challenging video reasoning benchmarks including VideoMME, TempCompass, VideoMMMU, MMVU, VSI-Bench, and PerceptionTest demonstrate consistent improvements over strong reinforcement learning baselines across multiple large multimodal models. Our results highlight the role of structured spatiotemporal exploration as a principled mechanism for stabilizing multimodal reinforcement learning and improving video reasoning performance.