Towards Differentially Private Reinforcement Learning with General Function Approximation

TL;DR

This paper provides the first theoretical guarantees for differentially private online reinforcement learning with general function approximation, achieving regret bounds comparable to linear cases and introducing new complexity measures.

Contribution

It extends differential privacy guarantees to general function approximation in RL, with novel regret analysis and complexity measures, surpassing prior tabular and linear results.

Findings

Regret scales as O(K^{3/5}) under differential privacy.

First regret bound for online RL with batch updates based on coverability.

Identifies gaps in recent private RL linear approximation results.

Abstract

We present the first theoretical guarantees for differentially private online reinforcement learning (RL) with general function approximation, extending beyond prior work restricted to tabular and linear settings. Our approach combines a batched policy update scheme with the exponential mechanism, together with a novel regret analysis. We show that, even under general function approximation, the regret in the model-free setting under differential privacy matches the state of the art for the linear case, scaling as $\widetilde{O}(K^{3/5})$, where $K$ denotes the number of episodes. As an important by-product, we also establish the first regret bound for online RL with batch update that depends on the standard complexity measure of coverability, complementing existing results based on a newly introduced Eluder-Condition class. In addition, we uncover fundamental gaps in recent results for private RL with linear function approximation, thereby clarifying its landscape.