FedQP: Towards Accurate Federated Learning using Quadratic Programming Guided Mutation

TL;DR

FedQP introduces a quadratic programming-guided mutation strategy in federated learning to improve global model accuracy under data heterogeneity by directing mutations towards better generalization areas.

Contribution

The paper presents a novel quadratic programming approach to guide mutations in federated learning, enhancing model generalization over existing stochastic mutation methods.

Findings

Improved inference accuracy on multiple datasets.

Effective handling of data heterogeneity.

Guided mutation outperforms random mutation strategies.

Abstract

Due to the advantages of privacy-preserving, Federated Learning (FL) is widely used in distributed machine learning systems. However, existing FL methods suffer from low-inference performance caused by data heterogeneity. Specifically, due to heterogeneous data, the optimization directions of different local models vary greatly, making it difficult for the traditional FL method to get a generalized global model that performs well on all clients. As one of the state-of-the-art FL methods, the mutation-based FL method attempts to adopt a stochastic mutation strategy to guide the model training towards a well-generalized area (i.e., flat area in the loss landscape). Specifically, mutation allows the model to shift within the solution space, providing an opportunity to escape areas with poor generalization (i.e., sharp area). However, the stochastic mutation strategy easily results in diverse optimal directions of mutated models, which limits the performance of the existing mutation-based FL method. To achieve higher performance, this paper proposes a novel mutation-based FL approach named FedQP, utilizing a quadratic programming strategy to regulate the mutation directions wisely. By biasing the model mutation towards the direction of gradient update rather than traditional random mutation, FedQP can effectively guide the model to optimize towards a well-generalized area (i.e., flat area). Experiments on multiple well-known datasets show that our quadratic programming-guided mutation strategy effectively improves the inference accuracy of the global model in various heterogeneous data scenarios.