Client Selection for Federated Policy Optimization with Environment Heterogeneity

TL;DR

This paper studies federated policy optimization in reinforcement learning, deriving error bounds considering environment differences, and proposes a client selection method that improves performance in heterogeneous environments.

Contribution

It introduces a theoretical analysis of federated Approximate Policy Iteration considering environment heterogeneity and proposes a client selection algorithm to reduce approximation errors.

Findings

The proposed client selection algorithm outperforms other methods in various RL tasks.

Theoretical error bounds are derived for federated API considering environment heterogeneity.

Experimental results demonstrate improved learning efficiency and performance.

Abstract

The development of Policy Iteration (PI) has inspired many recent algorithms for Reinforcement Learning (RL), including several policy gradient methods that gained both theoretical soundness and empirical success on a variety of tasks. The theory of PI is rich in the context of centralized learning, but its study under the federated setting is still in the infant stage. This paper investigates the federated version of Approximate PI (API) and derives its error bound, taking into account the approximation error introduced by environment heterogeneity. We theoretically prove that a proper client selection scheme can reduce this error bound. Based on the theoretical result, we propose a client selection algorithm to alleviate the additional approximation error caused by environment heterogeneity. Experiment results show that the proposed algorithm outperforms other biased and unbiased client selection methods on the federated mountain car problem, the Mujoco Hopper problem, and the SUMO-based autonomous vehicle training problem by effectively selecting clients with a lower level of heterogeneity from the population distribution.