Simple Entity-Centric Questions Challenge Dense Retrievers

TL;DR

This paper reveals that current dense retrieval models struggle with entity-rich questions and proposes that improved passage encoders, rather than data augmentation, are needed for better generalization across diverse questions.

Contribution

The paper introduces EntityQuestions, a new dataset highlighting dense retrievers' limitations with entity-centric questions, and suggests that specialized encoders improve robustness.

Findings

Dense retrievers underperform sparse methods on EntityQuestions.

Data augmentation does not improve generalization to new entities.

Enhanced passage encoders facilitate better question adaptation.

Abstract

Open-domain question answering has exploded in popularity recently due to the success of dense retrieval models, which have surpassed sparse models using only a few supervised training examples. However, in this paper, we demonstrate current dense models are not yet the holy grail of retrieval. We first construct EntityQuestions, a set of simple, entity-rich questions based on facts from Wikidata (e.g., "Where was Arve Furset born?"), and observe that dense retrievers drastically underperform sparse methods. We investigate this issue and uncover that dense retrievers can only generalize to common entities unless the question pattern is explicitly observed during training. We discuss two simple solutions towards addressing this critical problem. First, we demonstrate that data augmentation is unable to fix the generalization problem. Second, we argue a more robust passage encoder helps facilitate better question adaptation using specialized question encoders. We hope our work can shed light on the challenges in creating a robust, universal dense retriever that works well across different input distributions.