RepBERT: Contextualized Text Embeddings for First-Stage Retrieval

TL;DR

RepBERT introduces a novel method for first-stage retrieval using fixed-length contextualized embeddings, achieving state-of-the-art results with efficiency comparable to traditional bag-of-words approaches.

Contribution

It proposes RepBERT, a new approach that uses contextualized embeddings for retrieval, differing from exact term matching methods.

Findings

Achieves state-of-the-art results on MS MARCO Passage Ranking

Maintains efficiency comparable to bag-of-words methods

Demonstrates effectiveness of contextualized embeddings in retrieval

Abstract

Although exact term match between queries and documents is the dominant method to perform first-stage retrieval, we propose a different approach, called RepBERT, to represent documents and queries with fixed-length contextualized embeddings. The inner products of query and document embeddings are regarded as relevance scores. On MS MARCO Passage Ranking task, RepBERT achieves state-of-the-art results among all initial retrieval techniques. And its efficiency is comparable to bag-of-words methods.