Federated Ensemble-Directed Offline Reinforcement Learning

TL;DR

This paper introduces FEDORA, a federated ensemble-directed offline RL algorithm that effectively combines data from distributed agents, outperforming traditional methods in complex control tasks and real-world robotics applications.

Contribution

The paper proposes FEDORA, a novel ensemble-based federated offline RL algorithm that improves policy quality by distilling collective knowledge from distributed datasets.

Findings

FEDORA outperforms standard offline RL and federated approaches in continuous control tasks.

FEDORA achieves superior results on real-world datasets and a mobile robot.

The code and experiments are publicly available for reproducibility.

Abstract

We consider the problem of federated offline reinforcement learning (RL), a scenario under which distributed learning agents must collaboratively learn a high-quality control policy only using small pre-collected datasets generated according to different unknown behavior policies. Na\"{i}vely combining a standard offline RL approach with a standard federated learning approach to solve this problem can lead to poorly performing policies. In response, we develop the Federated Ensemble-Directed Offline Reinforcement Learning Algorithm (FEDORA), which distills the collective wisdom of the clients using an ensemble learning approach. We develop the FEDORA codebase to utilize distributed compute resources on a federated learning platform. We show that FEDORA significantly outperforms other approaches, including offline RL over the combined data pool, in various complex continuous control environments and real-world datasets. Finally, we demonstrate the performance of FEDORA in the real-world on a mobile robot. We provide our code and a video of our experiments at \url{https://github.com/DesikRengarajan/FEDORA}.