Information-Theoretic Opacity-Enforcement in Markov Decision Processes

TL;DR

This paper introduces information-theoretic methods to enforce opacity in Markov decision processes, balancing privacy of secrets with system performance using novel policy gradient algorithms.

Contribution

It develops primal-dual policy gradient algorithms for opacity enforcement in MDPs, leveraging message passing for efficient entropy gradient computation.

Findings

Algorithms achieve stable and fast convergence.

Effective enforcement of opacity in grid world example.

Balances privacy and system return constraints.

Abstract

The paper studies information-theoretic opacity, an information-flow privacy property, in a setting involving two agents: A planning agent who controls a stochastic system and an observer who partially observes the system states. The goal of the observer is to infer some secret, represented by a random variable, from its partial observations, while the goal of the planning agent is to make the secret maximally opaque to the observer while achieving a satisfactory total return. Modeling the stochastic system using a Markov decision process, two classes of opacity properties are considered -- Last-state opacity is to ensure that the observer is uncertain if the last state is in a specific set and initial-state opacity is to ensure that the observer is unsure of the realization of the initial state. As the measure of opacity, we employ the Shannon conditional entropy capturing the information about the secret revealed by the observable. Then, we develop primal-dual policy gradient methods for opacity-enforcement planning subject to constraints on total returns. We propose novel algorithms to compute the policy gradient of entropy for each observation, leveraging message passing within the hidden Markov models. This gradient computation enables us to have stable and fast convergence. We demonstrate our solution of opacity-enforcement control through a grid world example.