Towards User-level Private Reinforcement Learning with Human Feedback

TL;DR

This paper introduces AUP-RLHF, a novel framework that enhances user-level privacy in reinforcement learning with human feedback, balancing privacy protection and utility in language model alignment tasks.

Contribution

It proposes a new user-level label differential privacy method for RLHF and develops the AUP-RLHF algorithm with theoretical guarantees and improved performance.

Findings

AUP-RLHF guarantees user-level privacy with better utility.

The algorithm outperforms baselines in sentiment and summarization tasks.

Achieves a superior privacy-utility trade-off.

Abstract

Reinforcement Learning with Human Feedback (RLHF) has emerged as an influential technique, enabling the alignment of large language models (LLMs) with human preferences. Despite the promising potential of RLHF, how to protect user preference privacy has become a crucial issue. Most previous work has focused on using differential privacy (DP) to protect the privacy of individual data. However, they have concentrated primarily on item-level privacy protection and have unsatisfactory performance for user-level privacy, which is more common in RLHF. This study proposes a novel framework, AUP-RLHF, which integrates user-level label DP into RLHF. We first show that the classical random response algorithm, which achieves an acceptable performance in item-level privacy, leads to suboptimal utility when in the user-level settings. We then establish a lower bound for the user-level label DP-RLHF and develop the AUP-RLHF algorithm, which guarantees $(\varepsilon, \delta)$ user-level privacy and achieves an improved estimation error. Experimental results show that AUP-RLHF outperforms existing baseline methods in sentiment generation and summarization tasks, achieving a better privacy-utility trade-off.