Grep-BiasIR: A Dataset for Investigating Gender Representation-Bias in Information Retrieval Results
Klara Krieg, Emilia Parada-Cabaleiro, Gertraud Medicus, Oleg, Lesota, Markus Schedl, Navid Rekabsaz

TL;DR
Grep-BiasIR is a new dataset designed to study gender bias in information retrieval results, containing gender-sensitive queries and variations of relevant documents to analyze societal biases.
Contribution
The paper introduces Grep-BiasIR, a thoroughly-audited dataset for investigating gender representation bias in IR systems, covering diverse gender-related topics with multiple document variations.
Findings
Dataset enables analysis of gender bias in IR results
Supports research on societal impact of retrieval biases
Provides a benchmark for bias detection methods
Abstract
The provided contents by information retrieval (IR) systems can reflect the existing societal biases and stereotypes. Such biases in retrieval results can lead to further establishing and strengthening stereotypes in society and also in the systems. To facilitate the studies of gender bias in the retrieval results of IR systems, we introduce Gender Representation-Bias for Information Retrieval (Grep-BiasIR), a novel thoroughly-audited dataset consisting of 118 bias-sensitive neutral search queries. The set of queries covers a wide range of gender-related topics, for which a biased representation of genders in the search result can be considered as socially problematic. Each query is accompanied with one relevant and one non-relevant document, where the document is also provided in three variations of female, male, and neutral. The dataset is available at…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsNames, Identity, and Discrimination Research
