FedHQL: Federated Heterogeneous Q-Learning

TL;DR

FedHQL introduces a novel federated reinforcement learning framework that enables heterogeneous agents with different architectures to learn collaboratively, overcoming the limitations of previous homogeneous assumptions and improving sample efficiency.

Contribution

The paper proposes FedHQL, a new algorithm for federated reinforcement learning with heterogeneous agents, addressing the challenge of different policy architectures in distributed settings.

Findings

FedHQL improves sample efficiency for heterogeneous agents.

The method effectively handles diverse policy parameterizations.

Empirical results demonstrate enhanced learning performance.

Abstract

Federated Reinforcement Learning (FedRL) encourages distributed agents to learn collectively from each other's experience to improve their performance without exchanging their raw trajectories. The existing work on FedRL assumes that all participating agents are homogeneous, which requires all agents to share the same policy parameterization (e.g., network architectures and training configurations). However, in real-world applications, agents are often in disagreement about the architecture and the parameters, possibly also because of disparate computational budgets. Because homogeneity is not given in practice, we introduce the problem setting of Federated Reinforcement Learning with Heterogeneous And bLack-box agEnts (FedRL-HALE). We present the unique challenges this new setting poses and propose the Federated Heterogeneous Q-Learning (FedHQL) algorithm that principally addresses these challenges. We empirically demonstrate the efficacy of FedHQL in boosting the sample efficiency of heterogeneous agents with distinct policy parameterization using standard RL tasks.