Deceptive Sequential Decision-Making via Regularized Policy Optimization

TL;DR

This paper introduces a framework for autonomous systems to deceive adversaries by actively misrepresenting their reward functions through three novel regularization strategies, maintaining high performance while misleading observers.

Contribution

It proposes three new deception strategies—diversionary, targeted, and equivocal—for policy optimization in Markov decision processes to mislead adversaries using inverse reinforcement learning.

Findings

All deception strategies successfully mislead adversaries.

Deceptive policies retain at least 98% of optimal reward.

The framework is validated in multi-agent scenarios.

Abstract

Autonomous systems are increasingly expected to operate in the presence of adversaries, though adversaries may infer sensitive information simply by observing a system. Therefore, present a deceptive sequential decision-making framework that not only conceals sensitive information, but actively misleads adversaries about it. We model autonomous systems as Markov decision processes, with adversaries using inverse reinforcement learning to recover reward functions. To counter them, we present three regularization strategies for policy synthesis problems that actively deceive an adversary about a system's reward. ``Diversionary deception'' leads an adversary to draw any false conclusion about the system's reward function. ``Targeted deception'' leads an adversary to draw a specific false conclusion about the system's reward function. ``Equivocal deception'' leads an adversary to infer that the real reward and a false reward both explain the system's behavior. We show how each form of deception can be implemented in policy optimization problems and analytically bound the loss in total accumulated reward induced by deception. Next, we evaluate these developments in a multi-agent setting. We show that diversionary, targeted, and equivocal deception all steer the adversary to false beliefs while still attaining a total accumulated reward that is at least 98% of its optimal, non-deceptive value.