RAVine: Reality-Aligned Evaluation for Agentic Search

TL;DR

RAVine is a new evaluation framework for agentic search that aligns better with realistic user scenarios, accurately assesses iterative search processes, and considers efficiency, thereby advancing the development of intelligent search systems.

Contribution

RAVine introduces a reality-aligned evaluation method for agentic search, focusing on multi-point queries, iterative process assessment, and fine-grained accuracy, which addresses limitations of existing benchmarks.

Findings

Benchmarking models with RAVine reveals new insights into their search behaviors.

RAVine's ground truth construction improves evaluation accuracy.

The framework highlights the importance of iterative process assessment in agentic search.

Abstract

Agentic search, as a more autonomous and adaptive paradigm of retrieval augmentation, is driving the evolution of intelligent search systems. However, existing evaluation frameworks fail to align well with the goals of agentic search. First, the complex queries commonly used in current benchmarks often deviate from realistic user search scenarios. Second, prior approaches tend to introduce noise when extracting ground truth for end-to-end evaluations, leading to distorted assessments at a fine-grained level. Third, most current frameworks focus solely on the quality of final answers, neglecting the evaluation of the iterative process inherent to agentic search. To address these limitations, we propose RAVine -- a Reality-Aligned eValuation framework for agentic LLMs with search. RAVine targets multi-point queries and long-form answers that better reflect user intents, and introduces an attributable ground truth construction strategy to enhance the accuracy of fine-grained evaluation. Moreover, RAVine examines model's interaction with search tools throughout the iterative process, and accounts for factors of efficiency. We benchmark a series of models using RAVine and derive several insights, which we hope will contribute to advancing the development of agentic search systems. The code and datasets are available at https://github.com/SwordFaith/RAVine.