Contrastive Fine-tuning Improves Robustness for Neural Rankers

TL;DR

This paper introduces a contrastive fine-tuning method for neural rankers that enhances their robustness to noisy inputs, domain shifts, and query variations by leveraging a contrastive loss alongside standard ranking objectives.

Contribution

The paper proposes a novel contrastive fine-tuning approach that improves neural rankers' robustness to out-of-domain data and query perturbations, outperforming data augmentation methods.

Findings

Improved robustness to query reformulations and noise.

Enhanced zero-shot transfer performance.

Outperforms data augmentation in robustness.

Abstract

The performance of state-of-the-art neural rankers can deteriorate substantially when exposed to noisy inputs or applied to a new domain. In this paper, we present a novel method for fine-tuning neural rankers that can significantly improve their robustness to out-of-domain data and query perturbations. Specifically, a contrastive loss that compares data points in the representation space is combined with the standard ranking loss during fine-tuning. We use relevance labels to denote similar/dissimilar pairs, which allows the model to learn the underlying matching semantics across different query-document pairs and leads to improved robustness. In experiments with four passage ranking datasets, the proposed contrastive fine-tuning method obtains improvements on robustness to query reformulations, noise perturbations, and zero-shot transfer for both BERT and BART based rankers. Additionally, our experiments show that contrastive fine-tuning outperforms data augmentation for robustifying neural rankers.