Multimodal Reinforcement Learning with Adaptive Verifier for AI Agents

TL;DR

This paper introduces Argos, an adaptive verifier that enhances multimodal reinforcement learning by providing fine-grained, task-specific rewards, leading to state-of-the-art results and reduced reward hacking.

Contribution

The paper presents Argos, a novel reward agent that evaluates multiple aspects of reasoning, improving training and robustness of multimodal AI agents.

Findings

Achieves state-of-the-art results on multiple agentic tasks.

Reduces reward hacking in multimodal reinforcement learning.

Online verification prevents agents from collapsing to ungrounded solutions.

Abstract

Agentic reasoning models trained with multimodal reinforcement learning (MMRL) have become increasingly capable, yet they are almost universally optimized using sparse, outcome-based rewards computed based on the final answers. Richer rewards computed from the reasoning tokens can improve learning significantly by providing more fine-grained guidance. However, it is challenging to compute more informative rewards in MMRL beyond those based on outcomes since different samples may require different scoring functions and teacher models may provide noisy reward signals too. In this paper, we introduce the Argos (Agentic Reward for Grounded & Objective Scoring), a principled reward agent to train multimodal reasoning models for agentic tasks. For each sample, Argos selects from a pool of teacher-model derived and rule-based scoring functions to simultaneously evaluate: (i) final response accuracy, (ii) spatiotemporal localization of referred entities and actions, and (iii) the quality of the reasoning process. We find that by leveraging our agentic verifier across both SFT data curation and RL training, our model achieves state-of-the-art results across multiple agentic tasks such as spatial reasoning, visual hallucination as well as robotics and embodied AI benchmarks. Critically, we demonstrate that just relying on SFT post-training on highly curated reasoning data is insufficient, as agents invariably collapse to ungrounded solutions during RL without our online verification. We also show that our agentic verifier can help to reduce reward-hacking in MMRL. Finally, we also provide a theoretical justification for the effectiveness of Argos through the concept of pareto-optimality.