FOVA: Offline Federated Reinforcement Learning with Mixed-Quality Data

TL;DR

FOVA is a novel offline federated reinforcement learning framework that uses a vote mechanism to identify high-quality actions, improving performance and stability when dealing with mixed-quality data across clients.

Contribution

This paper introduces FOVA, a vote-based offline FRL method that effectively handles mixed-quality data and enhances policy learning through advantage-weighted regression.

Findings

FOVA outperforms existing methods on benchmark tasks.

Theoretical analysis confirms strict policy improvement.

FOVA demonstrates significant robustness to data quality variation.

Abstract

Offline Federated Reinforcement Learning (FRL), a marriage of federated learning and offline reinforcement learning, has attracted increasing interest recently. Albeit with some advancement, we find that the performance of most existing offline FRL methods drops dramatically when provided with mixed-quality data, that is, the logging behaviors (offline data) are collected by policies with varying qualities across clients. To overcome this limitation, this paper introduces a new vote-based offline FRL framework, named FOVA. It exploits a \emph{vote mechanism} to identify high-return actions during local policy evaluation, alleviating the negative effect of low-quality behaviors from diverse local learning policies. Besides, building on advantage-weighted regression (AWR), we construct consistent local and global training objectives, significantly enhancing the efficiency and stability of FOVA. Further, we conduct an extensive theoretical analysis and rigorously show that the policy learned by FOVA enjoys strict policy improvement over the behavioral policy. Extensive experiments corroborate the significant performance gains of our proposed algorithm over existing baselines on widely used benchmarks.