Attention-based Fault-tolerant Approach for Multi-agent Reinforcement Learning Systems

TL;DR

This paper introduces FT-Attn, an attention-based fault-tolerant algorithm for multi-agent reinforcement learning that enhances robustness against faulty or malicious agents, improving coordination in noisy and complex environments.

Contribution

The paper proposes a novel attention-based fault-tolerant method that enables multi-agent systems to effectively communicate and adapt despite adversarial or faulty agents.

Findings

FT-Attn outperforms previous methods in complex environments.

It adapts well to noisy environments without parameter tuning.

It handles scenarios requiring multiple correct observations simultaneously.

Abstract

The aim of multi-agent reinforcement learning systems is to provide interacting agents with the ability to collaboratively learn and adapt to the behavior of other agents. In many real-world applications, the agents can only acquire a partial view of the world. However, in realistic settings, one or more agents that show arbitrarily faulty or malicious behavior may suffice to let the current coordination mechanisms fail. In this paper, we study a practical scenario considering the security issues in the presence of agents with arbitrarily faulty or malicious behavior. Under these circumstances, learning an optimal policy becomes particularly challenging, even in the unrealistic case that an agent's policy can be made conditional upon all other agents' observations. To overcome these difficulties, we present an Attention-based Fault-Tolerant (FT-Attn) algorithm which selects correct and relevant information for each agent at every time-step. The multi-head attention mechanism enables the agents to learn effective communication policies through experience concurrently to the action policies. Empirical results have shown that FT-Attn beats previous state-of-the-art methods in some complex environments and can adapt to various kinds of noisy environments without tuning the complexity of the algorithm. Furthermore, FT-Attn can effectively deal with the complex situation where an agent needs to reach multiple agents' correct observation at the same time.