ExcluIR: Exclusionary Neural Information Retrieval

TL;DR

This paper introduces ExcluIR, a new resource and benchmark for exclusionary retrieval, revealing that current models struggle with such queries but can be improved, especially with generative approaches.

Contribution

The paper presents the first dedicated benchmark and training data for exclusionary retrieval, advancing research in understanding and improving models' handling of exclusionary queries.

Findings

Existing models struggle with exclusionary queries

Training data improves model performance but gaps remain

Generative models handle exclusionary queries better

Abstract

Exclusion is an important and universal linguistic skill that humans use to express what they do not want. However, in information retrieval community, there is little research on exclusionary retrieval, where users express what they do not want in their queries. In this work, we investigate the scenario of exclusionary retrieval in document retrieval for the first time. We present ExcluIR, a set of resources for exclusionary retrieval, consisting of an evaluation benchmark and a training set for helping retrieval models to comprehend exclusionary queries. The evaluation benchmark includes 3,452 high-quality exclusionary queries, each of which has been manually annotated. The training set contains 70,293 exclusionary queries, each paired with a positive document and a negative document. We conduct detailed experiments and analyses, obtaining three main observations: (1) Existing retrieval models with different architectures struggle to effectively comprehend exclusionary queries; (2) Although integrating our training data can improve the performance of retrieval models on exclusionary retrieval, there still exists a gap compared to human performance; (3) Generative retrieval models have a natural advantage in handling exclusionary queries. To facilitate future research on exclusionary retrieval, we share the benchmark and evaluation scripts on \url{https://github.com/zwh-sdu/ExcluIR}.