ICA: Information-Aware Credit Assignment for Visually Grounded Long-Horizon Information-Seeking Agents

TL;DR

This paper introduces a visual-native search framework with an information-aware credit assignment method to improve reinforcement learning agents' ability to locate relevant information in web environments, overcoming noise and sparse rewards.

Contribution

The paper proposes ICA, a novel post-hoc credit assignment technique leveraging visual snapshots and layout cues, integrated with a GRPO training pipeline for better web search performance.

Findings

Outperforms text-based baselines on multiple benchmarks.

Leverages visual layout cues to localize salient evidence.

Alleviates credit assignment bottleneck in web environments.

Abstract

Despite the strong performance achieved by reinforcement learning-trained information-seeking agents, learning in open-ended web environments remains severely constrained by low signal-to-noise feedback. Text-based parsers often discard layout semantics and introduce unstructured noise, while long-horizon training typically relies on sparse outcome rewards that obscure which retrieval actions actually matter. We propose a visual-native search framework that represents webpages as visual snapshots, allowing agents to leverage layout cues to quickly localize salient evidence and suppress distractors. To learn effectively from these high-dimensional observations, we introduce Information-Aware Credit Assignment (ICA), a post-hoc method that estimates each retrieved snapshot's contribution to the final outcome via posterior analysis and propagates dense learning signals back to key search turns. Integrated with a GRPO-based training pipeline, our approach consistently outperforms text-based baselines on diverse information-seeking benchmarks, providing evidence that visual snapshot grounding with information-level credit assignment alleviates the credit-assignment bottleneck in open-ended web environments. The code and datasets will be released in https://github.com/pc-inno/ICA_MM_deepsearch.git.