FORLER: Federated Offline Reinforcement Learning with Q-Ensemble and Actor Rectification

TL;DR

FORLER introduces a novel federated offline reinforcement learning framework that combines Q-ensemble aggregation and actor rectification, improving robustness and performance in heterogeneous, low-quality data environments.

Contribution

It proposes a new method integrating server Q-ensemble aggregation with device actor rectification, addressing policy pollution and computational constraints in offline federated RL.

Findings

Outperforms strong baselines across various data qualities.

Provides theoretical guarantees for safe policy improvement.

Effectively mitigates policy pollution in heterogeneous data settings.

Abstract

In Internet-of-Things systems, federated learning has advanced online reinforcement learning (RL) by enabling parallel policy training without sharing raw data. However, interacting with real environments online can be risky and costly, motivating offline federated RL (FRL), where local devices learn from fixed datasets. Despite its promise, offline FRL may break down under low-quality, heterogeneous data. Offline RL tends to get stuck in local optima, and in FRL, one device's suboptimal policy can degrade the aggregated model, i.e., policy pollution. We present FORLER, combining Q-ensemble aggregation on the server with actor rectification on devices. The server robustly merges device Q-functions to curb policy pollution and shift heavy computation off resource-constrained hardware without compromising privacy. Locally, actor rectification enriches policy gradients via a zeroth-order search for high-Q actions plus a bespoke regularizer that nudges the policy toward them. A $\delta$-periodic strategy further reduces local computation. We theoretically provide safe policy improvement performance guarantees. Extensive experiments show FORLER consistently outperforms strong baselines under varying data quality and heterogeneity.