Federated Reinforcement Learning in Heterogeneous Environments

TL;DR

This paper introduces a robust federated reinforcement learning framework for heterogeneous environments, proposing a new global objective, a novel algorithm FedRQ, and extensions to continuous spaces, with extensive empirical validation.

Contribution

It presents a new federated RL framework with a robust global objective, a convergent tabular algorithm FedRQ, and an extension to continuous spaces using expectile loss, enhancing robustness and applicability.

Findings

FedRQ converges asymptotically to an optimal policy.

The robust FRL framework outperforms existing algorithms in diverse environments.

Extensions to continuous spaces enable broader applicability.

Abstract

We investigate a Federated Reinforcement Learning with Environment Heterogeneity (FRL-EH) framework, where local environments exhibit statistical heterogeneity. Within this framework, agents collaboratively learn a global policy by aggregating their collective experiences while preserving the privacy of their local trajectories. To better reflect real-world scenarios, we introduce a robust FRL-EH framework by presenting a novel global objective function. This function is specifically designed to optimize a global policy that ensures robust performance across heterogeneous local environments and their plausible perturbations. We propose a tabular FRL algorithm named FedRQ and theoretically prove its asymptotic convergence to an optimal policy for the global objective function. Furthermore, we extend FedRQ to environments with continuous state space through the use of expectile loss, addressing the key challenge of minimizing a value function over a continuous subset of the state space. This advancement facilitates the seamless integration of the principles of FedRQ with various Deep Neural Network (DNN)-based RL algorithms. Extensive empirical evaluations validate the effectiveness and robustness of our FRL algorithms across diverse heterogeneous environments, consistently achieving superior performance over the existing state-of-the-art FRL algorithms.