Federated style aware transformer aggregation of representations

TL;DR

FedSTAR is a federated learning framework that disentangles style and content representations, uses attention-based prototype aggregation for personalization, and reduces communication costs while improving robustness in heterogeneous data environments.

Contribution

Introduces FedSTAR, a style-aware federated learning method that combines content-style disentanglement with attention-based prototype aggregation for enhanced personalization.

Findings

Improves personalization in federated learning with heterogeneous data.

Reduces communication overhead by exchanging compact prototypes.

Enhances robustness without increasing communication costs.

Abstract

Personalized Federated Learning (PFL) faces persistent challenges, including domain heterogeneity from diverse client data, data imbalance due to skewed participation, and strict communication constraints. Traditional federated learning often lacks personalization, as a single global model cannot capture client-specific characteristics, leading to biased predictions and poor generalization, especially for clients with highly divergent data distributions. To address these issues, we propose FedSTAR, a style-aware federated learning framework that disentangles client-specific style factors from shared content representations. FedSTAR aggregates class-wise prototypes using a Transformer-based attention mechanism, allowing the server to adaptively weight client contributions while preserving personalization. Furthermore, by exchanging compact prototypes and style vectors instead of full model parameters, FedSTAR significantly reduces communication overhead. Experimental results demonstrate that combining content-style disentanglement with attention-driven prototype aggregation improves personalization and robustness in heterogeneous environments without increasing communication cost.