Near-Optimal Reinforcement Learning with Shuffle Differential Privacy

TL;DR

This paper introduces SDP-PE, a novel reinforcement learning algorithm under the shuffle differential privacy model, achieving near-optimal regret bounds and balancing privacy with learning efficiency in networked systems.

Contribution

It presents the first policy elimination-based RL algorithm under the shuffle model, combining a new batching schedule and forgetting mechanism for improved privacy-utility trade-offs.

Findings

Achieves near-optimal regret bounds under shuffle DP.

Outperforms local DP models in utility while maintaining strong privacy.

Numerical experiments validate theoretical guarantees.

Abstract

Reinforcement learning (RL) is a powerful tool for sequential decision-making, but its application is often hindered by privacy concerns arising from its interaction data. This challenge is particularly acute in advanced networked systems, where learning from operational and user data can expose systems to privacy inference attacks. Existing differential privacy (DP) models for RL are often inadequate: the centralized model requires a fully trusted server, creating a single point of failure risk, while the local model incurs significant performance degradation that is unsuitable for many networked applications. This paper addresses this gap by leveraging the emerging shuffle model of privacy, an intermediate trust model that provides strong privacy guarantees without a centralized trust assumption. We present Shuffle Differentially Private Policy Elimination (SDP-PE), the first generic policy elimination-based algorithm for episodic RL under the shuffle model. Our method introduces a novel exponential batching schedule and a ``forgetting'' mechanism to balance the competing demands of privacy and learning performance. Our analysis shows that SDP-PE achieves a near-optimal regret bound, demonstrating a superior privacy-regret trade-off with utility comparable to the centralized model while significantly outperforming the local model. The numerical experiments also corroborate our theoretical results and demonstrate the effectiveness of SDP-PE. This work establishes the viability of the shuffle model for secure data-driven decision-making in networked systems.