Proxy Model-Guided Reinforcement Learning for Client Selection in Federated Recommendation

TL;DR

This paper introduces ProxyRL-FRS, a novel proxy model-guided reinforcement learning framework for client selection in federated recommendation systems, improving efficiency and personalization by estimating client contributions without costly training.

Contribution

The paper proposes ProxyRL-FRS, combining a lightweight proxy model with reinforcement learning to optimize client selection in federated recommendation, addressing data heterogeneity and evaluation cost issues.

Findings

ProxyRL-FRS outperforms baseline methods in recommendation accuracy.

The proxy model effectively estimates client contributions without extensive training.

Reinforcement learning improves client selection efficiency and diversity.

Abstract

Federated recommender systems have emerged as a promising privacy-preserving paradigm, enabling personalized recommendation services without exposing users' raw data. By keeping data local and relying on a central server to coordinate training across distributed clients, FedRSs protect user privacy while collaboratively learning global models. However, most existing FedRS frameworks adopt fully random client selection strategy in each training round, overlooking the statistical heterogeneity of user data arising from diverse preferences and behavior patterns, thereby resulting in suboptimal model performance. While some client selection strategies have been proposed in the broader federated learning literature, these methods are typically designed for generic tasks and fail to address the unique challenges of recommendation scenarios, such as expensive contribution evaluation due to the large number of clients, and sparse updates resulting from long-tail item distributions. To bridge this gap, we propose ProxyRL-FRS, a proxy model-guided reinforcement learning framework tailored for client selection in federated recommendation. Specifically, we first introduce ProxyNCF, a dual-branch model deployed on each client, which augments standard Neural Collaborative Filtering with an additional proxy model branch that provides lightweight contribution estimation, thus eliminating the need for expensive per-round local training traditionally required to evaluate a client's contribution. Furthermore, we design a staleness-aware SA reinforcement learning agent that selects clients based on the proxy-estimated contribution, and is guided by a reward function balancing recommendation accuracy and embedding staleness, thereby enriching the update coverage of item embeddings. Experiments conducted on public recommendation datasets demonstrate the effectiveness of ProxyRL-FRS.