Dense Retrievers Can Fail on Simple Queries: Revealing The Granularity Dilemma of Embeddings

TL;DR

This paper reveals that dense text encoders often fail to recognize fine-grained entities in simple queries, introduces a new dataset for evaluation, and demonstrates that targeted fine-tuning can significantly improve retrieval performance despite the granularity dilemma.

Contribution

The paper introduces CapRetrieval, a new dataset for evaluating fine-grained retrieval, and shows how fine-tuning with specific strategies enhances encoder performance, addressing the granularity dilemma.

Findings

Encoders struggle with fine-grained matching in simple queries.

Fine-tuning improves performance of smaller models beyond larger ones.

The granularity dilemma highlights a fundamental challenge in embedding semantics.

Abstract

This work stems from an observed limitation of text encoders: embeddings may not be able to recognize fine-grained entities or events within encoded semantics, resulting in failed retrieval even in simple cases. To examine such behaviors, we first introduce a new evaluation dataset, CapRetrieval, in which passages are image captions and queries are phrases targeting entity or event concepts in diverse forms. Zero-shot evaluation suggests that encoders often struggle with these fine-grained matching, regardless of training sources or model size. Aiming for enhancement, we proceed to finetune encoders with our proposed data generation strategies, enabling a small 0.1B encoder to outperform the state-of-the-art 7B model. Within this process, we further uncover the granularity dilemma, a challenge for embeddings to capture fine-grained salience while aligning with overall semantics. Our dataset, code and models in this work are publicly released at https://github.com/lxucs/CapRetrieval.