BiXSE: Improving Dense Retrieval via Probabilistic Graded Relevance Distillation

TL;DR

BiXSE introduces a pointwise training approach for dense retrieval that leverages LLM-generated graded relevance scores, enabling more nuanced supervision with less annotation and outperforming traditional contrastive methods.

Contribution

It presents BiXSE, a novel method that uses probabilistic graded relevance distillation with binary cross-entropy, reducing annotation costs and improving retrieval performance.

Findings

BiXSE outperforms softmax-based contrastive learning on multiple benchmarks.

BiXSE matches or exceeds pairwise ranking baselines with LLM-supervised data.

The method reduces annotation and compute costs while maintaining high performance.

Abstract

Neural sentence embedding models for dense retrieval typically rely on binary relevance labels, treating query-document pairs as either relevant or irrelevant. However, real-world relevance often exists on a continuum, and recent advances in large language models (LLMs) have made it feasible to scale the generation of fine-grained graded relevance labels. In this work, we propose BiXSE, a simple and effective pointwise training method that optimizes binary cross-entropy (BCE) over LLM-generated graded relevance scores. BiXSE interprets these scores as probabilistic targets, enabling granular supervision from a single labeled query-document pair per query. Unlike pairwise or listwise losses that require multiple annotated comparisons per query, BiXSE achieves strong performance with reduced annotation and compute costs by leveraging in-batch negatives. Extensive experiments across sentence embedding (MMTEB) and retrieval benchmarks (BEIR, TREC-DL) show that BiXSE consistently outperforms softmax-based contrastive learning (InfoNCE), and matches or exceeds strong pairwise ranking baselines when trained on LLM-supervised data. BiXSE offers a robust, scalable alternative for training dense retrieval models as graded relevance supervision becomes increasingly accessible.