A Replicability Study of XTR

TL;DR

This study replicates and extends the evaluation of the XTR retrieval algorithm, revealing its impact on retrieval efficiency and providing practical guidance for its deployment in modern retrieval systems.

Contribution

It replicates the original XTR algorithm and training modifications, extends evaluation to new training methods and engines, and clarifies XTR's practical utility.

Findings

XTR's effectiveness over ColBERT was not replicated under controlled conditions.

XTR training produces more discriminative centroid scores, improving IVF-based retrieval.

XTR training benefits are applicable beyond low-k regimes, aiding efficient retrieval.

Abstract

The XTR (conteXtual Token Retrieval) algorithm is a modification to ColBERT retrieval that avoids the costly step of fully gathering and reranking the candidates' embeddings by imputing their missing similarity scores from the initial token retrieval step. The original work proposes a modified training objective as necessary for effective XTR retrieval, arguing that standard ColBERT token scoring is unsuitable for imputation. In this paper, we replicate both the XTR retrieval algorithm and its modified training objective, and extend the evaluation to knowledge-distillation (KD) training and efficient retrieval engines (PLAID and WARP). We confirm the token-level matching characteristics claimed in the original work, but fail to replicate XTR's overall effectiveness advantage over ColBERT under a controlled comparison. We further show that XTR's training modification has a concrete mechanistic consequence for modern retrieval engines: by flattening ColBERT's characteristically peaked token score distribution, XTR training yields more discriminative centroid scores and thus more efficient IVF-based retrieval under PLAID and WARP. The utility of XTR training is therefore not limited to the low-$k'$ regime originally studied, but extends to any deployment setting where IVF-based engines are used. These findings offer practitioners concrete guidance on how and when to use XTR as their multi-vector retriever.