RaCT: Ranking-aware Chain-of-Thought Optimization for LLMs

TL;DR

This paper introduces RaCT, a novel approach combining Chain-of-Thought prompting with a two-stage training pipeline to improve LLMs' text reranking performance while preserving their general reasoning abilities.

Contribution

The paper presents a new methodology that integrates Chain-of-Thought prompting with supervised fine-tuning and ranking optimization, enhancing reranking accuracy without degrading general capabilities.

Findings

Outperforms state-of-the-art models like RankZephyr on TREC datasets.

Maintains robust performance across diverse reasoning tasks on MMLU.

Achieves significant improvements in ranking metrics such as nDCG.

Abstract

In information retrieval, large language models (LLMs) have demonstrated remarkable potential in text reranking tasks by leveraging their sophisticated natural language understanding and advanced reasoning capabilities. However, conventional supervised fine-tuning approaches for specializing LLMs in ranking tasks often lead to significant degradation of the models' general-purpose abilities. To address this fundamental challenge, this paper presents a novel methodology that strategically combines Chain-of-Thought (CoT) prompting techniques with an innovative two-stage training pipeline consisting of Supervised Fine-Tuning followed by Ranking Preference Optimization (SFT-RPO). The Chain-of-Thought prompting component encourages models to explicitly articulate their reasoning process during ranking decisions, creating a transparent pathway from query-document analysis to final ranking scores while maintaining analytical capabilities throughout fine-tuning. Extensive experimental evaluations on the TREC Deep Learning datasets demonstrate that our proposed method achieves superior performance compared to existing state-of-the-art models, including RankZephyr, showing consistent improvements across multiple evaluation metrics such as normalized Discounted Cumulative Gain (nDCG). Most significantly, comprehensive assessments on the Massive Multitask Language Understanding (MMLU) benchmark reveal that our method successfully maintains robust performance across diverse reasoning tasks, providing strong empirical evidence for effective retention of general-purpose capabilities through strategic fine-tuning while achieving specialized performance improvements in text reranking.