Provably Robust Federated Reinforcement Learning

TL;DR

This paper identifies vulnerabilities in existing Byzantine-robust federated reinforcement learning methods against a new Normalized attack and proposes an ensemble approach that provably defends against such attacks, improving robustness.

Contribution

It introduces a novel Normalized attack on FRL and develops an ensemble method that guarantees robustness against this and other known attacks.

Findings

Normalized attack significantly disrupts existing methods

Ensemble approach effectively defends against poisoning attacks

Experimental results confirm robustness of the proposed method

Abstract

Federated reinforcement learning (FRL) allows agents to jointly learn a global decision-making policy under the guidance of a central server. While FRL has advantages, its decentralized design makes it prone to poisoning attacks. To mitigate this, Byzantine-robust aggregation techniques tailored for FRL have been introduced. Yet, in our work, we reveal that these current Byzantine-robust techniques are not immune to our newly introduced Normalized attack. Distinct from previous attacks that targeted enlarging the distance of policy updates before and after an attack, our Normalized attack emphasizes on maximizing the angle of deviation between these updates. To counter these threats, we develop an ensemble FRL approach that is provably secure against both known and our newly proposed attacks. Our ensemble method involves training multiple global policies, where each is learnt by a group of agents using any foundational aggregation rule. These well-trained global policies then individually predict the action for a specific test state. The ultimate action is chosen based on a majority vote for discrete action systems or the geometric median for continuous ones. Our experimental results across different settings show that the Normalized attack can greatly disrupt non-ensemble Byzantine-robust methods, and our ensemble approach offers substantial resistance against poisoning attacks.