Federated Unbiased Learning to Rank

TL;DR

This paper introduces FedIPS, a federated learning algorithm for unbiased learning to rank that preserves user privacy by learning from local interactions and removing position bias using click propensities.

Contribution

It proposes FedIPS, a novel federated approach for unbiased learning to rank that operates on-device and addresses data scarcity and privacy concerns.

Findings

FedIPS outperforms baseline methods on Yahoo and Istella datasets.

FedIPS is robust across various position bias levels.

The approach effectively removes position bias using click propensities.

Abstract

Unbiased Learning to Rank (ULTR) studies the problem of learning a ranking function based on biased user interactions. In this framework, ULTR algorithms have to rely on a large amount of user data that are collected, stored, and aggregated by central servers. In this paper, we consider an on-device search setting, where users search against their personal corpora on their local devices, and the goal is to learn a ranking function from biased user interactions. Due to privacy constraints, users' queries, personal documents, results lists, and raw interaction data will not leave their devices, and ULTR has to be carried out via Federated Learning (FL). Directly applying existing ULTR algorithms on users' devices could suffer from insufficient training data due to the limited amount of local interactions. To address this problem, we propose the FedIPS algorithm, which learns from user interactions on-device under the coordination of a central server and uses click propensities to remove the position bias in user interactions. Our evaluation of FedIPS on the Yahoo and Istella datasets shows that FedIPS is robust over a range of position biases.