CoCoFL: Communication- and Computation-Aware Federated Learning via Partial NN Freezing and Quantization

TL;DR

CoCoFL is a federated learning method that maintains full neural network structures on all devices by freezing and quantizing layers, improving fairness and efficiency across heterogeneous devices.

Contribution

It introduces a novel approach to federated learning that preserves full neural networks on devices through selective freezing and quantization, enhancing fairness and resource utilization.

Findings

Improves fairness among heterogeneous devices in federated learning.

Reduces communication and computation costs via layer freezing and quantization.

Achieves higher overall model accuracy with resource-aware training.

Abstract

Devices participating in federated learning (FL) typically have heterogeneous communication, computation, and memory resources. However, in synchronous FL, all devices need to finish training by the same deadline dictated by the server. Our results show that training a smaller subset of the neural network (NN) at constrained devices, i.e., dropping neurons/filters as proposed by state of the art, is inefficient, preventing these devices to make an effective contribution to the model. This causes unfairness w.r.t the achievable accuracies of constrained devices, especially in cases with a skewed distribution of class labels across devices. We present a novel FL technique, CoCoFL, which maintains the full NN structure on all devices. To adapt to the devices' heterogeneous resources, CoCoFL freezes and quantizes selected layers, reducing communication, computation, and memory requirements, whereas other layers are still trained in full precision, enabling to reach a high accuracy. Thereby, CoCoFL efficiently utilizes the available resources on devices and allows constrained devices to make a significant contribution to the FL system, increasing fairness among participants (accuracy parity) and significantly improving the final accuracy of the model.