Distilling Dense Representations for Ranking using Tightly-Coupled Teachers

TL;DR

This paper introduces a knowledge distillation method that simplifies dense ranking models, significantly reducing latency and storage while maintaining high effectiveness by tightly coupling teacher and student models.

Contribution

The paper proposes a novel distillation approach that tightly couples teacher and student models, enabling efficient dense retrieval with improved speed and storage efficiency.

Findings

Improves query latency and reduces storage requirements of ColBERT.

Achieves near cross-encoder effectiveness with much faster retrieval.

Tightly-coupled distillation enhances representation quality.

Abstract

We present an approach to ranking with dense representations that applies knowledge distillation to improve the recently proposed late-interaction ColBERT model. Specifically, we distill the knowledge from ColBERT's expressive MaxSim operator for computing relevance scores into a simple dot product, thus enabling single-step ANN search. Our key insight is that during distillation, tight coupling between the teacher model and the student model enables more flexible distillation strategies and yields better learned representations. We empirically show that our approach improves query latency and greatly reduces the onerous storage requirements of ColBERT, while only making modest sacrifices in terms of effectiveness. By combining our dense representations with sparse representations derived from document expansion, we are able to approach the effectiveness of a standard cross-encoder reranker using BERT that is orders of magnitude slower.