Reproducing NevIR: Negation in Neural Information Retrieval

TL;DR

This paper reproduces and extends NevIR, analyzing how neural IR models handle negation, revealing that even advanced models struggle with negation and highlighting the need for better negation understanding in IR systems.

Contribution

It replicates NevIR's findings, evaluates new state-of-the-art models, and introduces ExcluIR to assess negation generalization, providing insights into model performance and data distribution effects.

Findings

Listwise LLM re-rankers outperform others but still lag behind humans.

Fine-tuning on one dataset does not improve performance on another.

Cross-encoders and listwise LLMs perform reasonably across negation tasks.

Abstract

Negation is a fundamental aspect of human communication, yet it remains a challenge for Language Models (LMs) in Information Retrieval (IR). Despite the heavy reliance of modern neural IR systems on LMs, little attention has been given to their handling of negation. In this study, we reproduce and extend the findings of NevIR, a benchmark study that revealed most IR models perform at or below the level of random ranking when dealing with negation. We replicate NevIR's original experiments and evaluate newly developed state-of-the-art IR models. Our findings show that a recently emerging category-listwise Large Language Model (LLM) re-rankers-outperforms other models but still underperforms human performance. Additionally, we leverage ExcluIR, a benchmark dataset designed for exclusionary queries with extensive negation, to assess the generalisability of negation understanding. Our findings suggest that fine-tuning on one dataset does not reliably improve performance on the other, indicating notable differences in their data distributions. Furthermore, we observe that only cross-encoders and listwise LLM re-rankers achieve reasonable performance across both negation tasks.