FedORGP: Guiding Heterogeneous Federated Learning with Orthogonality Regularization on Global Prototypes

TL;DR

FedORGP introduces orthogonality regularization on global prototypes in heterogeneous federated learning, significantly improving prototype separation and model performance under data and model heterogeneity.

Contribution

The paper proposes FedORGP, a novel algorithm that enhances prototype separation in Heterogeneous Federated Learning using orthogonality regularization.

Findings

Achieves up to 10.12% accuracy improvement over baselines.

Provides theoretical convergence proof under non-convex conditions.

Outperforms seven state-of-the-art methods in heterogeneous scenarios.

Abstract

Federated Learning (FL) has emerged as an essential framework for distributed machine learning, especially with its potential for privacy-preserving data processing. However, existing FL frameworks struggle to address statistical and model heterogeneity, which severely impacts model performance. While Heterogeneous Federated Learning (HtFL) introduces prototype-based strategies to address the challenges, current approaches face limitations in achieving optimal separation of prototypes. This paper presents FedORGP, a novel HtFL algorithm designed to improve global prototype separation through orthogonality regularization, which not only encourages intra-class prototype similarity but also significantly expands the inter-class angular separation. With the guidance of the global prototype, each client keeps its embeddings aligned with the corresponding prototype in the feature space, promoting directional independence that integrates seamlessly with the cross-entropy (CE) loss. We provide theoretical proof of FedORGP's convergence under non-convex conditions. Extensive experiments demonstrate that FedORGP outperforms seven state-of-the-art baselines, achieving up to 10.12\% accuracy improvement in scenarios where statistical and model heterogeneity coexist.