FGR-ColBERT: Identifying Fine-Grained Relevance Tokens During Retrieval

TL;DR

FGR-ColBERT enhances document retrieval by integrating fine-grained relevance signals from LLMs directly into the model, achieving high accuracy with significantly reduced computational costs.

Contribution

It introduces FGR-ColBERT, a modified retrieval model that incorporates LLM-derived relevance cues, improving token-level relevance detection efficiently.

Findings

FGR-ColBERT achieves a token-level F1 of 64.5, surpassing larger models.

It maintains 99% of retrieval effectiveness of state-of-the-art models.

The method incurs only about 1.12x latency overhead.

Abstract

Document retrieval identifies relevant documents but does not provide fine-grained evidence cues, such as specific relevant spans. A possible solution is to apply an LLM after retrieval; however, this introduces significant computational overhead and limits practical deployment. We propose FGR-ColBERT, a modification of ColBERT retrieval model that integrates fine-grained relevance signals distilled from an LLM directly into the retrieval function. Experiments on MS MARCO show that FGR-ColBERT (110M) achieves a token-level F1 of 64.5, exceeding the 62.8 of Gemma 2 (27B), despite being approximately 245 times smaller. At the same time, it preserves retrieval effectiveness (99% relative Recall@50) and remains efficient, incurring only a ~1.12x latency overhead compared to the original ColBERT.