Does Reasoning Make Search More Fair? Comparing Fairness in Reasoning and Non-Reasoning Rerankers
Saron Samuel, Benjamin Van Durme, Eugene Yang

TL;DR
This study systematically compares fairness in reasoning and non-reasoning rerankers using the TREC 2022 dataset, finding that reasoning rerankers neither improve nor harm fairness, with fairness gaps persisting across demographics.
Contribution
First systematic comparison of fairness between reasoning and non-reasoning rerankers, revealing that reasoning models do not inherently affect fairness.
Findings
Fairness metric (AWRF) remained stable across models.
Relevance varied significantly, but fairness did not.
Fairness gaps persist for geographic attributes.
Abstract
While reasoning rerankers, such as Rank1, have demonstrated strong abilities in improving ranking relevance, it is unclear how they perform on other retrieval qualities such as fairness. We conduct the first systematic comparison of fairness between reasoning and non-reasoning rerankers. Using the TREC 2022 Fair Ranking Track dataset, we evaluate six reranking models across multiple retrieval settings and demographic attributes. Our findings demonstrate reasoning neither improve nor harm fairness compared to non-reasoning approaches. Our fairness metric, Attention-Weighted Rank Fairness (AWRF) remained stable (0.33-0.35) across all models, even as relevance varies substantially (nDCG 0.247-1.000). Demographic breakdown analysis revealed fairness gaps for geographic attributes regardless of model architecture. These results indicate that future work in specializing reasoning models to be…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsEthics and Social Impacts of AI · Computational and Text Analysis Methods · Mobile Crowdsensing and Crowdsourcing
