Towards Federated RLHF with Aggregated Client Preference for LLMs

TL;DR

This paper introduces federated reinforcement learning with human feedback (RLHF) for large language models, enabling preference learning without sharing sensitive data, and demonstrates improved content quality through novel aggregation methods.

Contribution

It proposes FedBis and FedBiscuit, innovative federated RLHF techniques that encode and aggregate client preferences, addressing heterogeneity and privacy concerns in preference-based LLM fine-tuning.

Findings

Significant improvement in professionalism of generated content.

First federated RLHF benchmark with heterogeneous preferences.

Effective handling of preference heterogeneity and reward hacking.

Abstract

Reinforcement learning with human feedback (RLHF) fine-tunes a pretrained large language model (LLM) using user preference data, enabling it to generate content aligned with human preferences. However, due to privacy concerns, users may be reluctant to share sensitive preference data. To address this, we propose utilizing Federated Learning (FL) techniques, allowing large-scale preference collection from diverse real-world users without requiring them to transmit data to a central server. Our federated RLHF methods (i.e., FedBis and FedBiscuit) encode each client's preferences into binary selectors and aggregate them to capture common preferences. In particular, FedBiscuit overcomes key challenges, such as preference heterogeneity and reward hacking, through innovative solutions like grouping clients with similar preferences to reduce heterogeneity and using multiple binary selectors to enhance LLM output quality. To evaluate the performance of the proposed methods, we establish the first federated RLHF benchmark with a heterogeneous human preference dataset. Experimental results show that by integrating the LLM with aggregated client preferences, FedBis and FedBiscuit significantly enhance the professionalism and readability of the generated content.