Reward-Free Attacks in Multi-Agent Reinforcement Learning

TL;DR

This paper explores reward-free attacks in multi-agent reinforcement learning, demonstrating that an attacker can strategically subvert a victim agent by maximizing policy entropy solely through observation, without access to reward information.

Contribution

It introduces a reward-free exploration algorithm that enables an attacker to maximize the victim's policy entropy, revealing a new vulnerability in multi-agent systems.

Findings

Attacker can subvert agents by maximizing policy entropy.

Reward-free attacks are feasible without reward access.

Policy entropy maximization effectively compromises victim agents.

Abstract

We investigate how effective an attacker can be when it only learns from its victim's actions, without access to the victim's reward. In this work, we are motivated by the scenario where the attacker wants to behave strategically when the victim's motivations are unknown. We argue that one heuristic approach an attacker can use is to maximize the entropy of the victim's policy. The policy is generally not obfuscated, which implies it may be extracted simply by passively observing the victim. We provide such a strategy in the form of a reward-free exploration algorithm that maximizes the attacker's entropy during the exploration phase, and then maximizes the victim's empirical entropy during the planning phase. In our experiments, the victim agents are subverted through policy entropy maximization, implying an attacker might not need access to the victim's reward to succeed. Hence, reward-free attacks, which are based only on observing behavior, show the feasibility of an attacker to act strategically without knowledge of the victim's motives even if the victim's reward information is protected.