FedGRPO: Privately Optimizing Foundation Models with Group-Relative Rewards from Domain Client

TL;DR

FedGRPO introduces a privacy-preserving federated learning framework that uses group-relative rewards and expert selection to improve foundation model performance efficiently across diverse domains.

Contribution

It reformulates federated foundation model training as a reinforcement learning process, reducing privacy risks and communication costs while enhancing accuracy.

Findings

Outperforms baseline methods in accuracy across multiple domain tasks.

Reduces communication overhead compared to traditional FedFMs.

Maintains privacy by exchanging reward signals instead of data or model updates.

Abstract

One important direction of Federated Foundation Models (FedFMs) is leveraging data from small client models to enhance the performance of a large server-side foundation model. Existing methods based on model level or representation level knowledge transfer either require expensive local training or incur high communication costs and introduce unavoidable privacy risks. We reformulate this problem as a reinforcement learning style evaluation process and propose FedGRPO, a privacy preserving framework comprising two modules. The first module performs competence-based expert selection by building a lightweight confidence graph from auxiliary data to identify the most suitable clients for each question. The second module leverages the "Group Relative" concept from the Group Relative Policy Optimization (GRPO) framework by packaging each question together with its solution rationale into candidate policies, dispatching these policies to a selected subset of expert clients, and aggregating solely the resulting scalar reward signals via a federated group-relative loss function. By exchanging reward values instead of data or model updates, FedGRPO reduces privacy risk and communication overhead while enabling parallel evaluation across heterogeneous devices. Empirical results on diverse domain tasks demonstrate that FedGRPO achieves superior downstream accuracy and communication efficiency compared to conventional FedFMs baselines.