Preserving Expert-Level Privacy in Offline Reinforcement Learning

TL;DR

This paper introduces a privacy-preserving offline reinforcement learning method that guarantees expert privacy through differential privacy, while maintaining strong empirical performance across complex environments.

Contribution

It proposes a novel consensus-based differentially private offline RL approach compatible with existing algorithms, with rigorous privacy guarantees and demonstrated empirical effectiveness.

Findings

Achieves differential privacy guarantees in offline RL.

Maintains strong empirical performance on complex environments.

Outperforms baseline methods in privacy-preserving settings.

Abstract

The offline reinforcement learning (RL) problem aims to learn an optimal policy from historical data collected by one or more behavioural policies (experts) by interacting with an environment. However, the individual experts may be privacy-sensitive in that the learnt policy may retain information about their precise choices. In some domains like personalized retrieval, advertising and healthcare, the expert choices are considered sensitive data. To provably protect the privacy of such experts, we propose a novel consensus-based expert-level differentially private offline RL training approach compatible with any existing offline RL algorithm. We prove rigorous differential privacy guarantees, while maintaining strong empirical performance. Unlike existing work in differentially private RL, we supplement the theory with proof-of-concept experiments on classic RL environments featuring large continuous state spaces, demonstrating substantial improvements over a natural baseline across multiple tasks.