O$^2$-Searcher: A Searching-based Agent Model for Open-Domain Open-Ended Question Answering

TL;DR

O$^2$-Searcher is a reinforcement learning-based search agent designed for open-domain, open-ended question answering, effectively handling diverse question types by dynamically acquiring knowledge and adapting answer strategies, outperforming existing models.

Contribution

The paper introduces O$^2$-Searcher, a novel search agent that uses reinforcement learning and local search environments to improve open-ended question answering in open domains.

Findings

Outperforms leading LLM agents on the O$^2$-QA benchmark.

Achieves state-of-the-art results on various closed-ended QA benchmarks.

Uses only a 3B parameter model to surpass larger models.

Abstract

Large Language Models (LLMs), despite their advancements, are fundamentally limited by their static parametric knowledge, hindering performance on tasks requiring open-domain up-to-date information. While enabling LLMs to interact with external knowledge environments is a promising solution, current efforts primarily address closed-end problems. Open-ended questions, which characterized by lacking a standard answer or providing non-unique and diverse answers, remain underexplored. To bridge this gap, we present O$^2$-Searcher, a novel search agent leveraging reinforcement learning to effectively tackle both open-ended and closed-ended questions in the open domain. O$^2$-Searcher leverages an efficient, locally simulated search environment for dynamic knowledge acquisition, effectively decoupling the external world knowledge from model's sophisticated reasoning processes. It employs a unified training mechanism with meticulously designed reward functions, enabling the agent to identify problem types and adapt different answer generation strategies. Furthermore, to evaluate performance on complex open-ended tasks, we construct O$^2$-QA, a high-quality benchmark featuring 300 manually curated, multi-domain open-ended questions with associated web page caches. Extensive experiments show that O$^2$-Searcher, using only a 3B model, significantly surpasses leading LLM agents on O$^2$-QA. It also achieves SOTA results on various closed-ended QA benchmarks against similarly-sized models, while performing on par with much larger ones.