Preserving the Privacy of Reward Functions in MDPs through Deception

TL;DR

This paper introduces a deception-based approach to protect reward function privacy in MDPs against IRL observers, outperforming existing methods by providing stronger privacy guarantees and maintaining expected rewards.

Contribution

It presents a novel RL-based planning algorithm using simulation for privacy preservation, addressing gaps in differential privacy methods against IRL-based inference.

Findings

Outperforms previous privacy-preserving methods in benchmarks

Demonstrates significant privacy leaks in existing dissimulation techniques

Provides theoretical guarantees on expected rewards

Abstract

Preserving the privacy of preferences (or rewards) of a sequential decision-making agent when decisions are observable is crucial in many physical and cybersecurity domains. For instance, in wildlife monitoring, agents must allocate patrolling resources without revealing animal locations to poachers. This paper addresses privacy preservation in planning over a sequence of actions in MDPs, where the reward function represents the preference structure to be protected. Observers can use Inverse RL (IRL) to learn these preferences, making this a challenging task. Current research on differential privacy in reward functions fails to ensure guarantee on the minimum expected reward and offers theoretical guarantees that are inadequate against IRL-based observers. To bridge this gap, we propose a novel approach rooted in the theory of deception. Deception includes two models: dissimulation (hiding the truth) and simulation (showing the wrong). Our first contribution theoretically demonstrates significant privacy leaks in existing dissimulation-based methods. Our second contribution is a novel RL-based planning algorithm that uses simulation to effectively address these privacy concerns while ensuring a guarantee on the expected reward. Experiments on multiple benchmark problems show that our approach outperforms previous methods in preserving reward function privacy.