Mitigating Preference Leakage via Strict Estimator Separation for Normative Generative Ranking

TL;DR

This paper introduces a leakage-free two-judge framework for cultural relevance ranking in Generative Information Retrieval, effectively reducing circularity and preference leakage, and demonstrates its effectiveness on new benchmarks and datasets.

Contribution

It formalizes cultural relevance as a within-query ranking task and proposes a strict separation of supervision and evaluation to mitigate preference leakage.

Findings

A dense bi-encoder distilled from Judge-B-supervised Cross-Encoder is highly effective.

The leakage-free evaluation framework outperforms classical baselines.

Distilled models align well with human norms on curated datasets.

Abstract

In Generative Information Retrieval (GenIR), the bottleneck has shifted from generation to the selection of candidates, particularly for normative criteria such as cultural relevance. Current LLM-as-a-Judge evaluations often suffer from circularity and preference leakage, where overlapping supervision and evaluation models inflate performance. We address this by formalising cultural relevance as a within-query ranking task and introducing a leakage-free two-judge framework that strictly separates supervision (Judge B) from evaluation (Judge A). On a new benchmark of 33,052 (NGR-33k) culturally grounded stories, we find that while classical baselines yield only modest gains, a dense bi-encoder distilled from a Judge-B-supervised Cross-Encoder is highly effective. Although the Cross-Encoder provides a strong supervision signal for distillation, the distilled BGE-M3 model substantially outperforms it under leakage-free Judge~A evaluation. We validate our framework on the human-curated Moral Stories dataset, showing strong alignment with human norms. Our results demonstrate that rigorous evaluator separation is a prerequisite for credible GenIR evaluation, proving that subtle cultural preferences can be distilled into efficient rankers without leakage.