Weaver: End-to-End Agentic System Training for Video Interleaved Reasoning

TL;DR

Weaver is an end-to-end multimodal reasoning system that dynamically utilizes tools and reinforcement learning to improve performance on complex video reasoning tasks involving long videos.

Contribution

Weaver introduces a novel end-to-end trainable system that dynamically invokes tools and employs reinforcement learning for improved video reasoning.

Findings

Enhanced performance on complex video reasoning benchmarks

Effective integration of tool invocation and reinforcement learning

Improved reasoning over long videos

Abstract

Video reasoning constitutes a comprehensive assessment of a model's capabilities, as it demands robust perceptual and interpretive skills, thereby serving as a means to explore the boundaries of model performance. While recent research has leveraged text-centric Chain-of-Thought reasoning to augment these capabilities, such approaches frequently suffer from representational mismatch and restricted by limited perceptual acuity. To address these limitations, we propose Weaver, a novel, end-to-end trainable multimodal reasoning agentic system. Weaver empowers its policy model to dynamically invoke diverse tools throughout the reasoning process, enabling progressive acquisition of crucial visual cues and construction of authentic multimodal reasoning trajectories. Furthermore, we integrate a reinforcement learning algorithm to allow the system to freely explore strategies for employing and combining these tools with trajectory-free data. Extensive experiments demonstrate that our system, Weaver, enhances performance on several complex video reasoning benchmarks, particularly those involving long videos.