Measuring and Addressing Indexical Bias in Information Retrieval
Caleb Ziems, William Held, Jane Dwivedi-Yu, Diyi Yang

TL;DR
This paper introduces the PAIR framework and DUO metric to automatically measure indexical bias in IR systems, demonstrating their effectiveness through extensive evaluation and human validation.
Contribution
It presents the first automatic bias metric for IR, along with a comprehensive evaluation on diverse systems and a human study validating its predictive power.
Findings
DUO effectively measures indexical bias.
The framework predicts bias impact on reader opinions.
Evaluation on 8 IR systems shows variability in bias levels.
Abstract
Information Retrieval (IR) systems are designed to deliver relevant content, but traditional systems may not optimize rankings for fairness, neutrality, or the balance of ideas. Consequently, IR can often introduce indexical biases, or biases in the positional order of documents. Although indexical bias can demonstrably affect people's opinion, voting patterns, and other behaviors, these issues remain understudied as the field lacks reliable metrics and procedures for automatically measuring indexical bias. Towards this end, we introduce the PAIR framework, which supports automatic bias audits for ranked documents or entire IR systems. After introducing DUO, the first general-purpose automatic bias metric, we run an extensive evaluation of 8 IR systems on a new corpus of 32k synthetic and 4.7k natural documents, with 4k queries spanning 1.4k controversial issue topics. A human…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
Taxonomy
TopicsMulti-Criteria Decision Making · Information Retrieval and Search Behavior
