Careful Queries, Credible Results: Teaching RAG Models Advanced Web Search Tools with Reinforcement Learning

TL;DR

This paper introduces WebFilter, a novel retrieval-augmented generation framework that uses reinforcement learning to generate source-restricted queries and filter unreliable web content, significantly improving answer quality and retrieval accuracy in noisy web environments.

Contribution

WebFilter is the first RAG framework to integrate source-restricted query generation with content filtering using reinforcement learning, addressing misinformation and underutilization of web tools.

Findings

WebFilter outperforms existing RAG methods on multiple benchmarks.

It enhances answer quality and retrieval precision.

The approach effectively mitigates misinformation in web-based retrieval.

Abstract

Retrieval-Augmented Generation (RAG) enhances large language models (LLMs) by integrating up-to-date external knowledge, yet real-world web environments present unique challenges. These limitations manifest as two key challenges: pervasive misinformation in the web environment, which introduces unreliable or misleading content that can degrade retrieval accuracy, and the underutilization of web tools, which, if effectively employed, could enhance query precision and help mitigate this noise, ultimately improving the retrieval results in RAG systems. To address these issues, we propose WebFilter, a novel RAG framework that generates source-restricted queries and filters out unreliable content. This approach combines a retrieval filtering mechanism with a behavior- and outcome-driven reward strategy, optimizing both query formulation and retrieval outcomes. Extensive experiments demonstrate that WebFilter improves answer quality and retrieval precision, outperforming existing RAG methods on both in-domain and out-of-domain benchmarks.