Modeling Ranking Properties with In-Context Learning

TL;DR

This paper introduces an in-context learning approach for ranking models that uses example demonstrations to balance relevance, diversity, and fairness without task-specific training, enabling flexible control over ranking behavior.

Contribution

The work presents a novel ICL method for IR ranking that eliminates the need for training by leveraging demonstration engineering to control multiple objectives.

Findings

Effective control over ranking trade-offs demonstrated

Applicable to multiple IR datasets and objectives

No explicit optimization required for desired behaviors

Abstract

While standard IR models are mainly designed to optimize relevance, real-world search often needs to balance additional objectives such as diversity and fairness. These objectives depend on inter-document interactions and are commonly addressed using post-hoc heuristics or supervised learning methods, which require task-specific training for each ranking scenario and dataset. In this work, we propose an in-context learning (ICL) approach that eliminates the need for such training. Instead, our method relies on a small number of example rankings that demonstrate the desired trade-offs between objectives for past queries similar to the current input. We evaluate our approach on four IR test collections to investigate multiple auxiliary objectives: group fairness (TREC Fairness), polarity diversity (Touch\'e), and topical diversity (TREC Deep Learning 2019/2020). We empirically validate that our method enables control over ranking behavior through demonstration engineering, allowing nuanced behavioral adjustments without explicit optimization.