University of Washington at TREC 2020 Fairness Ranking Track

TL;DR

The University of Washington's FATE group participated in the 2020 TREC Fairness Ranking Track, developing modules to incorporate author identity dimensions like gender and location into retrieval and re-ranking tasks, with mixed performance results.

Contribution

This work introduces a modular approach to integrating author identity features into fairness-aware retrieval and re-ranking systems for TREC data.

Findings

Performed above average in retrieval tasks

Performed below par in re-ranking tasks

Demonstrated modular extraction of identity features

Abstract

InfoSeeking Lab's FATE (Fairness Accountability Transparency Ethics) group at University of Washington participated in 2020 TREC Fairness Ranking Track. This report describes that track, assigned data and tasks, our group definitions, and our results. Our approach to bringing fairness in retrieval and re-ranking tasks with Semantic Scholar data was to extract various dimensions of author identity. These dimensions included gender and location. We developed modules for these extractions in a way that allowed us to plug them in for either of the tasks as needed. After trying different combinations of relative weights assigned to relevance, gender, and location information, we chose five runs for retrieval and five runs for re-ranking tasks. The results showed that our runs performed below par for re-ranking task, but above average for retrieval.