FairDiverse: A Comprehensive Toolkit for Fair and Diverse Information Retrieval Algorithms

TL;DR

FairDiverse is an open-source toolkit that standardizes the evaluation of fairness and diversity algorithms in information retrieval, enabling consistent benchmarking across multiple IR tasks and models.

Contribution

It introduces a comprehensive, extensible framework for evaluating fairness and diversity algorithms in IR, addressing inconsistencies in prior evaluation methods.

Findings

Supports 28 algorithms across 16 models

Covers search and recommendation tasks

Facilitates fair comparison and development

Abstract

In modern information retrieval (IR). achieving more than just accuracy is essential to sustaining a healthy ecosystem, especially when addressing fairness and diversity considerations. To meet these needs, various datasets, algorithms, and evaluation frameworks have been introduced. However, these algorithms are often tested across diverse metrics, datasets, and experimental setups, leading to inconsistencies and difficulties in direct comparisons. This highlights the need for a comprehensive IR toolkit that enables standardized evaluation of fairness- and diversity-aware algorithms across different IR tasks. To address this challenge, we present FairDiverse, an open-source and standardized toolkit. FairDiverse offers a framework for integrating fair and diverse methods, including pre-processing, in-processing, and post-processing techniques, at different stages of the IR pipeline. The toolkit supports the evaluation of 28 fairness and diversity algorithms across 16 base models, covering two core IR tasks (search and recommendation) thereby establishing a comprehensive benchmark. Moreover, FairDiverse is highly extensible, providing multiple APIs that empower IR researchers to swiftly develop and evaluate their own fairness and diversity aware models, while ensuring fair comparisons with existing baselines. The project is open-sourced and available on https://github.com/XuChen0427/FairDiverse.