ColBERT-Att: Late-Interaction Meets Attention for Enhanced Retrieval

TL;DR

ColBERT-Att enhances neural information retrieval by integrating attention mechanisms into late interaction models, improving recall accuracy across multiple benchmarks.

Contribution

This work introduces ColBERT-Att, a novel approach that explicitly incorporates attention weights into late interaction models for better relevance estimation.

Findings

Improved recall accuracy on MS-MARCO dataset

Enhanced performance on BEIR and LoTTE benchmarks

Effective integration of attention mechanisms in retrieval models

Abstract

Vector embeddings from pre-trained language models form a core component in Neural Information Retrieval systems across a multitude of knowledge extraction tasks. The paradigm of late interaction, introduced in ColBERT, demonstrates high accuracy along with runtime efficiency. However, the current formulation fails to take into account the attention weights of query and document terms, which intuitively capture the "importance" of similarities between them, that might lead to a better understanding of relevance between the queries and documents. This work proposes ColBERT-Att, to explicitly integrate attention mechanism into the late interaction framework for enhanced retrieval performance. Empirical evaluation of ColBERT-Att depicts improvements in recall accuracy on MS-MARCO as well as on a wide range of BEIR and LoTTE benchmark datasets.