Beyond Content Relevance: Evaluating Instruction Following in Retrieval Models

TL;DR

This paper evaluates how well retrieval models follow user instructions beyond content relevance, introducing a new benchmark and metrics to assess responsiveness to instructions, revealing current models still struggle with instruction compliance.

Contribution

It presents InfoSearch, a new benchmark with six document attributes, and introduces SICR and WISE metrics to evaluate instruction-following in retrieval models, highlighting areas for improvement.

Findings

Larger and fine-tuned models perform better but still lack full instruction compliance.

Most retrieval models do not fully adhere to user instructions despite advancements.

The new benchmark and metrics provide a comprehensive way to evaluate instruction following.

Abstract

Instruction-following capabilities in LLMs have progressed significantly, enabling more complex user interactions through detailed prompts. However, retrieval systems have not matched these advances, most of them still relies on traditional lexical and semantic matching techniques that fail to fully capture user intent. Recent efforts have introduced instruction-aware retrieval models, but these primarily focus on intrinsic content relevance, which neglects the importance of customized preferences for broader document-level attributes. This study evaluates the instruction-following capabilities of various retrieval models beyond content relevance, including LLM-based dense retrieval and reranking models. We develop InfoSearch, a novel retrieval evaluation benchmark spanning six document-level attributes: Audience, Keyword, Format, Language, Length, and Source, and introduce novel metrics -- Strict Instruction Compliance Ratio (SICR) and Weighted Instruction Sensitivity Evaluation (WISE) to accurately assess the models' responsiveness to instructions. Our findings indicate that although fine-tuning models on instruction-aware retrieval datasets and increasing model size enhance performance, most models still fall short of instruction compliance.