Inducing Diversity in Differentiable Search Indexing

TL;DR

This paper introduces a method to train Differentiable Search Indexing models that balances relevance and diversity, improving information retrieval by integrating diversity directly into the training process without extra post-processing.

Contribution

We propose a novel training approach inspired by MMR that induces diversity in DSI models, enabling relevance and diversity to be learned simultaneously during training.

Findings

Achieves diversity without sacrificing relevance.

Outperforms naive DSI in relevance-diversity trade-offs.

Applicable to incremental DSI updates for fast, diverse retrieval.

Abstract

Differentiable Search Indexing (DSI) is a recent paradigm for information retrieval which uses a transformer-based neural network architecture as the document index to simplify the retrieval process. A differentiable index has many advantages enabling modifications, updates or extensions to the index. In this work, we explore balancing relevance and novel information content (diversity) for training DSI systems inspired by Maximal Marginal Relevance (MMR), and show the benefits of our approach over the naive DSI training. We present quantitative and qualitative evaluations of relevance and diversity measures obtained using our method on NQ320K and MSMARCO datasets in comparison to naive DSI. With our approach, it is possible to achieve diversity without any significant impact to relevance. Since we induce diversity while training DSI, the trained model has learned to diversify while being relevant. This obviates the need for a post-processing step to induce diversity in the recall set as typically performed using MMR. Our approach will be useful for Information Retrieval problems where both relevance and diversity are important such as in sub-topic retrieval. Our work can also be easily be extended to the incremental DSI settings which would enable fast updates to the index while retrieving a diverse recall set.