Adversarial Yet Cooperative: Multi-Perspective Reasoning in Retrieved-Augmented Language Models

TL;DR

This paper introduces ARR, a framework where reasoning and verification models collaboratively improve multi-step reasoning in retrieval-augmented language models through adversarial interaction and process-aware rewards.

Contribution

It presents a novel Reasoner-Verifier framework enabling multi-perspective, self-correcting reasoning without external scoring, advancing retrieval-augmented language model capabilities.

Findings

Improved reasoning accuracy on multiple benchmarks

Effective self-critique and correction in reasoning process

Enhanced reasoning fidelity without external rewards

Abstract

Recent advances in synergizing large reasoning models (LRMs) with retrieval-augmented generation (RAG) have shown promising results, yet two critical challenges remain: (1) reasoning models typically operate from a single, unchallenged perspective, limiting their ability to conduct deep, self-correcting reasoning over external documents, and (2) existing training paradigms rely excessively on outcome-oriented rewards, which provide insufficient signal for shaping the complex, multi-step reasoning process. To address these issues, we propose an Reasoner-Verifier framework named Adversarial Reasoning RAG (ARR). The Reasoner and Verifier engage in reasoning on retrieved evidence and critiquing each other's logic while being guided by process-aware advantage that requires no external scoring model. This reward combines explicit observational signals with internal model uncertainty to jointly optimize reasoning fidelity and verification rigor. Experiments on multiple benchmarks demonstrate the effectiveness of our method.