Collective Relevance Labeling for Passage Retrieval

TL;DR

This paper introduces a collective knowledge distillation method for passage retrieval that uses an efficient teacher model to produce high-quality relevance labels, improving ranking performance while being faster than existing methods.

Contribution

We propose a simple, efficient collective knowledge distillation approach that outperforms complex teachers in passage retrieval tasks, with significantly reduced training time.

Findings

Our method trains up to 8 times faster than state-of-the-art teachers.

It achieves better distillation of rankings compared to more complex models.

The approach is scalable and publicly available for use.

Abstract

Deep learning for Information Retrieval (IR) requires a large amount of high-quality query-document relevance labels, but such labels are inherently sparse. Label smoothing redistributes some observed probability mass over unobserved instances, often uniformly, uninformed of the true distribution. In contrast, we propose knowledge distillation for informed labeling, without incurring high computation overheads at evaluation time. Our contribution is designing a simple but efficient teacher model which utilizes collective knowledge, to outperform state-of-the-arts distilled from a more complex teacher model. Specifically, we train up to x8 faster than the state-of-the-art teacher, while distilling the rankings better. Our code is publicly available at https://github.com/jihyukkim-nlp/CollectiveKD