Online Distillation for Pseudo-Relevance Feedback

TL;DR

This paper introduces an online distillation method for pseudo-relevance feedback, enabling efficient lexical models to replicate neural re-ranking and improve document retrieval performance.

Contribution

It presents a novel online distillation approach that allows query-specific models to be distilled from neural re-ranking results for enhanced retrieval.

Findings

Online distilled models can effectively mimic neural re-ranking.

Distilled models improve retrieval by identifying missed relevant documents.

Approach outperforms traditional pseudo relevance feedback and hybrid methods.

Abstract

Model distillation has emerged as a prominent technique to improve neural search models. To date, distillation taken an offline approach, wherein a new neural model is trained to predict relevance scores between arbitrary queries and documents. In this paper, we explore a departure from this offline distillation strategy by investigating whether a model for a specific query can be effectively distilled from neural re-ranking results (i.e., distilling in an online setting). Indeed, we find that a lexical model distilled online can reasonably replicate the re-ranking of a neural model. More importantly, these models can be used as queries that execute efficiently on indexes. This second retrieval stage can enrich the pool of documents for re-ranking by identifying documents that were missed in the first retrieval stage. Empirically, we show that this approach performs favourably when compared with established pseudo relevance feedback techniques, dense retrieval methods, and sparse-dense ensemble "hybrid" approaches.