Experience Constrained Hierarchical Federated Reinforcement Learning for Large-scale UAV Teams in Hazardous Environments

TL;DR

This paper proposes EC-HFRL, a hierarchical federated reinforcement learning framework for UAV teams in hazardous environments, emphasizing experience reuse over participation levels for improved learning performance.

Contribution

It introduces a novel hierarchical federated RL approach that highlights the importance of experience reuse strategies over participation in experience-constrained settings.

Findings

Increasing participation does not necessarily improve learning performance.

Experience reuse strategy and gradient transition experiences are key to performance.

Empirical results show structure of learning signal outweighs federated aggregation effects.

Abstract

Conventional federated learning assumes that greater learner participation improves training performance, by leveraging abundant, independently generated local data. However, in federated reinforcement learning (FRL) for unmanned aerial vehicle (UAV) teams in hazardous environments where experience generation is severely constrained by safety considerations, energy limitations, and mission duration, this assumption may break. This work introduces Experience-Constrained Hierarchical Federated Reinforcement Learning (EC-HFRL), a framework in which clusters act as federated learning agents, while multiple intra-cluster learners represent parallel learning resources that reuse a shared experience pool. We show that increasing participation does not necessarily improve learning performance. Instead, learning performance is strongly associated with experience reuse strategy and the dominance of key analytically identified gradient transition experiences within a cluster. In particular, minibatch size primarily determines effective replay exposure, while higher intra-cluster participation increases reuse level. Empirical results demonstrate that the performance regimes are strongly associated with the structure of the learning signal, rather than federated aggregation effects, clarifying the limited and secondary role of learner participation in experience-constrained FRL.