ImpliRet: Benchmarking the Implicit Fact Retrieval Challenge

TL;DR

ImpliRet introduces a benchmark that evaluates document-side reasoning capabilities in retrieval systems, revealing significant challenges for current models in understanding implicit facts through complex relationships.

Contribution

The paper presents ImpliRet, a novel benchmark focusing on implicit fact retrieval in documents, shifting the reasoning challenge from query to document side, and evaluates various retrieval models on this task.

Findings

Current retrieval models perform poorly on implicit fact retrieval, with nDCG@10 only 14.91%.

Even long-context models like GPT-4-mini struggle, scoring only 55.54% with thirty documents.

Document-side reasoning remains a significant challenge for NLP retrieval systems.

Abstract

Retrieval systems are central to many NLP pipelines, but often rely on surface-level cues such as keyword overlap and lexical semantic similarity. To evaluate retrieval beyond these shallow signals, recent benchmarks introduce reasoning-heavy queries; however, they primarily shift the burden to query-side processing techniques -- like prompting or multi-hop retrieval -- that can help resolve complexity. In contrast, we present Impliret, a benchmark that shifts the reasoning challenge to document-side processing: The queries are simple, but relevance depends on facts stated implicitly in documents through temporal (e.g., resolving "two days ago"), arithmetic, and world knowledge relationships. We evaluate a range of sparse and dense retrievers, all of which struggle in this setting: the best nDCG@10 is only 14.91%. We also test whether long-context models can overcome this limitation. But even with a short context of only thirty documents, including the positive document, GPT-o4-mini scores only 55.54%, showing that document-side reasoning remains a challenge. Our codes are available at github.com/ZeinabTaghavi/IMPLIRET.