Stealthy Imitation: Reward-guided Environment-free Policy Stealing

TL;DR

This paper introduces Stealthy Imitation, a novel black-box attack method for stealing deep reinforcement learning policies without environment access, and proposes a countermeasure to defend against it.

Contribution

It presents the first environment-free, reward-guided policy stealing attack and a countermeasure, advancing the understanding of model theft vulnerabilities in RL systems.

Findings

Outperforms prior data-free policy stealing methods.

Matching attack query distribution to the victim's reduces imitation difficulty.

Countermeasure significantly diminishes attack effectiveness.

Abstract

Deep reinforcement learning policies, which are integral to modern control systems, represent valuable intellectual property. The development of these policies demands considerable resources, such as domain expertise, simulation fidelity, and real-world validation. These policies are potentially vulnerable to model stealing attacks, which aim to replicate their functionality using only black-box access. In this paper, we propose Stealthy Imitation, the first attack designed to steal policies without access to the environment or knowledge of the input range. This setup has not been considered by previous model stealing methods. Lacking access to the victim's input states distribution, Stealthy Imitation fits a reward model that allows to approximate it. We show that the victim policy is harder to imitate when the distribution of the attack queries matches that of the victim. We evaluate our approach across diverse, high-dimensional control tasks and consistently outperform prior data-free approaches adapted for policy stealing. Lastly, we propose a countermeasure that significantly diminishes the effectiveness of the attack.