P^3 Ranker: Mitigating the Gaps between Pre-training and Ranking Fine-tuning with Prompt-based Learning and Pre-finetuning

TL;DR

This paper introduces P^3 Ranker, a novel approach that reduces the gap between pre-training and ranking fine-tuning in language models by using prompt-based learning and pre-finetuning, leading to improved few-shot ranking performance.

Contribution

The paper proposes P^3 Ranker, combining prompt-based learning and pre-finetuning to better align pre-trained models with ranking tasks, addressing key mismatches.

Findings

P^3 Ranker outperforms baselines in few-shot ranking on MS MARCO and Robust04.

Prompt-based learning helps adapt models more effectively to ranking tasks.

Pre-finetuning enables models to acquire ranking-specific knowledge.

Abstract

Compared to other language tasks, applying pre-trained language models (PLMs) for search ranking often requires more nuances and training signals. In this paper, we identify and study the two mismatches between pre-training and ranking fine-tuning: the training schema gap regarding the differences in training objectives and model architectures, and the task knowledge gap considering the discrepancy between the knowledge needed in ranking and that learned during pre-training. To mitigate these gaps, we propose Pre-trained, Prompt-learned and Pre-finetuned Neural Ranker (P^3 Ranker). P^3 Ranker leverages prompt-based learning to convert the ranking task into a pre-training like schema and uses pre-finetuning to initialize the model on intermediate supervised tasks. Experiments on MS MARCO and Robust04 show the superior performances of P^3 Ranker in few-shot ranking. Analyses reveal that P^3 Ranker is able to better accustom to the ranking task through prompt-based learning and retrieve necessary ranking-oriented knowledge gleaned in pre-finetuning, resulting in data-efficient PLM adaptation. Our code is available at https://github.com/NEUIR/P3Ranker.