FedMOA: Federated GRPO for Personalized Reasoning LLMs under Heterogeneous Rewards

TL;DR

FedMOA introduces a federated reinforcement learning framework that enhances personalized reasoning in large language models by effectively handling heterogeneous rewards and multi-objective optimization, leading to improved accuracy and personalization.

Contribution

The paper presents FedMOA, a novel federated GRPO method with adaptive weighting and task-aware aggregation for multi-objective alignment under heterogeneous rewards.

Findings

Achieves up to 2.2% accuracy improvement over federated averaging.

Enhances global performance and personalization in reasoning tasks.

Effectively manages heterogeneous rewards and multi-objective optimization.

Abstract

Group Relative Policy Optimization (GRPO) has recently emerged as an effective approach for improving the reasoning capabilities of large language models through online multi-objective reinforcement learning. While personalization on private data is increasingly vital, traditional Reinforcement Learning (RL) alignment is often memory-prohibitive for on-device federated learning due to the overhead of maintaining a separate critic network. GRPO's critic-free architecture enables feasible on-device training, yet transitioning to a federated setting introduces systemic challenges: heterogeneous reward definitions, imbalanced multi-objective optimization, and high training costs. We propose FedMOA, a federated GRPO framework for multi-objective alignment under heterogeneous rewards. FedMOA stabilizes local training through an online adaptive weighting mechanism via hypergradient descent, which prioritizes primary reasoning as auxiliary objectives saturate. On the server side, it utilizes a task- and accuracy-aware aggregation strategy to prioritize high-quality updates. Experiments on mathematical reasoning and code generation benchmarks demonstrate that FedMOA consistently outperforms federated averaging, achieving accuracy gains of up to 2.2% while improving global performance, personalization, and multi-objective balance.