The Overlooked Role of Graded Relevance Thresholds in Multilingual Dense Retrieval

TL;DR

This paper investigates how the choice of relevance thresholds impacts multilingual dense retrieval performance, highlighting the importance of calibrating thresholds to improve effectiveness and reduce data needs.

Contribution

It demonstrates that optimal relevance thresholds vary across languages and tasks, and emphasizes threshold calibration as a crucial step in dense retrieval fine-tuning.

Findings

Optimal thresholds differ systematically across languages and tasks.

Proper threshold calibration enhances retrieval effectiveness.

Incorrect thresholds can degrade performance and increase annotation noise.

Abstract

Dense retrieval models are typically fine-tuned with contrastive learning objectives that require binary relevance judgments, even though relevance is inherently graded. We analyze how graded relevance scores and the threshold used to convert them into binary labels affect multilingual dense retrieval. Using a multilingual dataset with LLM-annotated relevance scores, we examine monolingual, multilingual mixture, and cross-lingual retrieval scenarios. Our findings show that the optimal threshold varies systematically across languages and tasks, often reflecting differences in resource level. A well-chosen threshold can improve effectiveness, reduce the amount of fine-tuning data required, and mitigate annotation noise, whereas a poorly chosen one can degrade performance. We argue that graded relevance is a valuable but underutilized signal for dense retrieval, and that threshold calibration should be treated as a principled component of the fine-tuning pipeline.