Near-Optimal Differentially Private Reinforcement Learning

TL;DR

This paper introduces a near-optimal differentially private reinforcement learning algorithm that achieves asymptotically optimal regret, matching non-private lower bounds, and introduces new techniques for privacy-preserving data release.

Contribution

It presents the first private RL algorithm with asymptotically optimal regret, achieving privacy for free as data size grows, and introduces novel privacy-preserving techniques.

Findings

Achieves regret matching non-private lower bounds for JDP.

Introduces new methods for privately releasing exploration bonuses.

Provides improved regret bounds for LDP case.

Abstract

Motivated by personalized healthcare and other applications involving sensitive data, we study online exploration in reinforcement learning with differential privacy (DP) constraints. Existing work on this problem established that no-regret learning is possible under joint differential privacy (JDP) and local differential privacy (LDP) but did not provide an algorithm with optimal regret. We close this gap for the JDP case by designing an $\epsilon$-JDP algorithm with a regret of $\widetilde{O}(\sqrt{SAH^2T}+S^2AH^3/\epsilon)$ which matches the information-theoretic lower bound of non-private learning for all choices of $\epsilon> S^{1.5}A^{0.5} H^2/\sqrt{T}$. In the above, $S$, $A$ denote the number of states and actions, $H$ denotes the planning horizon, and $T$ is the number of steps. To the best of our knowledge, this is the first private RL algorithm that achieves \emph{privacy for free} asymptotically as $T\rightarrow \infty$. Our techniques -- which could be of independent interest -- include privately releasing Bernstein-type exploration bonuses and an improved method for releasing visitation statistics. The same techniques also imply a slightly improved regret bound for the LDP case.