FedFormer: Contextual Federation with Attention in Reinforcement Learning

TL;DR

FedFormer introduces a Transformer-based attention mechanism for federated reinforcement learning, enabling more effective aggregation of multi-agent insights and outperforming traditional averaging methods in complex environments.

Contribution

This paper presents FedFormer, a novel federation strategy using Transformer Attention to contextually aggregate agent models, improving performance in multi-agent reinforcement learning.

Findings

FedFormer outperforms FedAvg and single-agent methods in Meta-World tasks.

Higher episodic returns achieved with FedFormer while maintaining privacy constraints.

Effectiveness increases with larger agent pools, unlike FedAvg.

Abstract

A core issue in multi-agent federated reinforcement learning is defining how to aggregate insights from multiple agents. This is commonly done by taking the average of each participating agent's model weights into one common model (FedAvg). We instead propose FedFormer, a novel federation strategy that utilizes Transformer Attention to contextually aggregate embeddings from models originating from different learner agents. In so doing, we attentively weigh the contributions of other agents with respect to the current agent's environment and learned relationships, thus providing a more effective and efficient federation. We evaluate our methods on the Meta-World environment and find that our approach yields significant improvements over FedAvg and non-federated Soft Actor-Critic single-agent methods. Our results compared to Soft Actor-Critic show that FedFormer achieves higher episodic return while still abiding by the privacy constraints of federated learning. Finally, we also demonstrate improvements in effectiveness with increased agent pools across all methods in certain tasks. This is contrasted by FedAvg, which fails to make noticeable improvements when scaled.