On the Sample Complexity of Differentially Private Policy Optimization

TL;DR

This paper provides a theoretical analysis of the sample complexity in differentially private policy optimization, addressing privacy challenges in reinforcement learning and offering insights into privacy costs.

Contribution

It formalizes a DP definition for PO and analyzes sample complexity of key algorithms under privacy constraints using a unified framework.

Findings

Privacy costs can appear as lower-order terms in sample complexity.

Systematic analysis of DP sample complexity for PG and NPG algorithms.

Insights into subtle privacy-related effects in policy optimization.

Abstract

Policy optimization (PO) is a cornerstone of modern reinforcement learning (RL), with diverse applications spanning robotics, healthcare, and large language model training. The increasing deployment of PO in sensitive domains, however, raises significant privacy concerns. In this paper, we initiate a theoretical study of differentially private policy optimization, focusing explicitly on its sample complexity. We first formalize an appropriate definition of differential privacy (DP) tailored to PO, addressing the inherent challenges arising from on-policy learning dynamics and the subtlety involved in defining the unit of privacy. We then systematically analyze the sample complexity of widely-used PO algorithms, including policy gradient (PG), natural policy gradient (NPG) and more, under DP constraints and various settings, via a unified framework. Our theoretical results demonstrate that privacy costs can often manifest as lower-order terms in the sample complexity, while also highlighting subtle yet important observations in private PO settings. These offer valuable practical insights for privacy-preserving PO algorithms.