Beyond [CLS] through Ranking by Generation

TL;DR

This paper revisits generative models for information retrieval, demonstrating that they can match the performance of modern discriminative models and highlighting the potential of unlikelihood training for IR tasks.

Contribution

It shows that generative models like GPT2 and BART are effective as rankers in IR, challenging the dominance of discriminative approaches and introducing unlikelihood loss for improved IR performance.

Findings

Generative models perform as well as state-of-the-art discriminative models in answer selection.

Unlikelihood loss enhances the effectiveness of generative IR models.

Revisiting generative frameworks offers a promising alternative in IR ranking methods.

Abstract

Generative models for Information Retrieval, where ranking of documents is viewed as the task of generating a query from a document's language model, were very successful in various IR tasks in the past. However, with the advent of modern deep neural networks, attention has shifted to discriminative ranking functions that model the semantic similarity of documents and queries instead. Recently, deep generative models such as GPT2 and BART have been shown to be excellent text generators, but their effectiveness as rankers have not been demonstrated yet. In this work, we revisit the generative framework for information retrieval and show that our generative approaches are as effective as state-of-the-art semantic similarity-based discriminative models for the answer selection task. Additionally, we demonstrate the effectiveness of unlikelihood losses for IR.