Inverse-Inverse Reinforcement Learning. How to Hide Strategy from an Adversarial Inverse Reinforcement Learner

TL;DR

This paper introduces a method for an agent to hide its true strategy from an adversarial inverse reinforcement learning attacker by deliberately choosing sub-optimal responses, enhancing privacy in strategic interactions.

Contribution

The paper proposes a novel inverse IRL approach that enables an agent to mask its strategy, including algorithms and theoretical analysis based on revealed preference theory.

Findings

The adversarial IRL algorithm can estimate strategies while controlling utility.

The proposed I-IRL method effectively spoofs the adversary’s IRL estimation.

Sample complexity results show robustness under noisy utility estimates.

Abstract

Inverse reinforcement learning (IRL) deals with estimating an agent's utility function from its actions. In this paper, we consider how an agent can hide its strategy and mitigate an adversarial IRL attack; we call this inverse IRL (I-IRL). How should the decision maker choose its response to ensure a poor reconstruction of its strategy by an adversary performing IRL to estimate the agent's strategy? This paper comprises four results: First, we present an adversarial IRL algorithm that estimates the agent's strategy while controlling the agent's utility function. Our second result for I-IRL result spoofs the IRL algorithm used by the adversary. Our I-IRL results are based on revealed preference theory in micro-economics. The key idea is for the agent to deliberately choose sub-optimal responses that sufficiently masks its true strategy. Third, we give a sample complexity result for our main I-IRL result when the agent has noisy estimates of the adversary specified utility function. Finally, we illustrate our I-IRL scheme in a radar problem where a meta-cognitive radar is trying to mitigate an adversarial target.