Does Reasoning Make Search More Fair? Comparing Fairness in Reasoning and Non-Reasoning Rerankers

TL;DR

This study systematically compares fairness in reasoning and non-reasoning rerankers using the TREC 2022 dataset, finding that reasoning rerankers neither improve nor harm fairness, with fairness gaps persisting across demographics.

Contribution

First systematic comparison of fairness between reasoning and non-reasoning rerankers, revealing that reasoning models do not inherently affect fairness.

Findings

Fairness metric (AWRF) remained stable across models.

Relevance varied significantly, but fairness did not.

Fairness gaps persist for geographic attributes.

Abstract

While reasoning rerankers, such as Rank1, have demonstrated strong abilities in improving ranking relevance, it is unclear how they perform on other retrieval qualities such as fairness. We conduct the first systematic comparison of fairness between reasoning and non-reasoning rerankers. Using the TREC 2022 Fair Ranking Track dataset, we evaluate six reranking models across multiple retrieval settings and demographic attributes. Our findings demonstrate reasoning neither improve nor harm fairness compared to non-reasoning approaches. Our fairness metric, Attention-Weighted Rank Fairness (AWRF) remained stable (0.33-0.35) across all models, even as relevance varies substantially (nDCG 0.247-1.000). Demographic breakdown analysis revealed fairness gaps for geographic attributes regardless of model architecture. These results indicate that future work in specializing reasoning models to be aware of fairness attributes could lead to improvements, as current implementations preserve the fairness characteristics of their input ranking.