RankT5: Fine-Tuning T5 for Text Ranking with Ranking Losses

TL;DR

This paper introduces RankT5, a novel T5-based model for text ranking that directly outputs scores and is fine-tuned with ranking losses, leading to improved performance and better zero-shot generalization.

Contribution

It proposes two T5-based ranking model structures and demonstrates their effectiveness with ranking losses, advancing sequence-to-sequence models for text ranking tasks.

Findings

Ranking models with ranking losses outperform classification-based models.

Listwise fine-tuning improves zero-shot out-of-domain ranking performance.

RankT5 achieves substantial performance gains on public datasets.

Abstract

Recently, substantial progress has been made in text ranking based on pretrained language models such as BERT. However, there are limited studies on how to leverage more powerful sequence-to-sequence models such as T5. Existing attempts usually formulate text ranking as classification and rely on postprocessing to obtain a ranked list. In this paper, we propose RankT5 and study two T5-based ranking model structures, an encoder-decoder and an encoder-only one, so that they not only can directly output ranking scores for each query-document pair, but also can be fine-tuned with "pairwise" or "listwise" ranking losses to optimize ranking performances. Our experiments show that the proposed models with ranking losses can achieve substantial ranking performance gains on different public text ranking data sets. Moreover, when fine-tuned with listwise ranking losses, the ranking model appears to have better zero-shot ranking performance on out-of-domain data sets compared to the model fine-tuned with classification losses.