Adaptive Guidance for Retrieval-Augmented Masked Diffusion Models

TL;DR

This paper introduces ARAM, a training-free adaptive guidance method for Masked Diffusion Models in retrieval-augmented generation, which dynamically adjusts guidance based on context reliability to enhance question-answering accuracy.

Contribution

The work presents a novel adaptive guidance framework for diffusion models that calibrates guidance during denoising based on context reliability, addressing retrieval conflicts.

Findings

ARAM improves QA performance over baseline RAG methods.

Dynamic guidance calibration enhances handling of noisy retrieved context.

Extensive experiments validate the effectiveness of ARAM across benchmarks.

Abstract

Retrieval-Augmented Generation (RAG) improves factual grounding by incorporating external knowledge into language model generation. However, when retrieved context is noisy, unreliable, or inconsistent with the model's parametric knowledge, it introduces retrieval-prior conflicts that can degrade generation quality. While this problem has been studied in autoregressive language models, it remains largely unexplored in diffusion-based language models, where the iterative denoising process introduces unique challenges for integrating retrieved context. In this work, we propose Adaptive Retrieval-Augmented Masked Diffusion (ARAM), a training-free adaptive guidance framework for Masked Diffusion Models (MDMs) in RAG settings. ARAM dynamically calibrates the guidance scale during denoising according to the Signal-to-Noise Ratio (SNR) of the distributional shift induced by retrieved context. Intuitively, the model strengthens guidance when the retrieved context provides reliable corrective evidence and suppresses it when the contextual signal is noisy or non-supportive. Extensive experiments on multiple knowledge-intensive QA benchmarks show that ARAM improves overall QA performance over competitive RAG baselines.