Federated Two-stage Learning with Sign-based Voting

TL;DR

This paper introduces a federated two-stage learning framework that reduces communication costs and enhances privacy by using a cut layer for local data representation and sign-based voting for model updates, suitable for general applications.

Contribution

It proposes a novel federated learning approach combining a cut layer and sign-based voting, improving efficiency and privacy over traditional methods.

Findings

Reduces communication overhead with sign-based SGD and majority voting.

Enhances privacy by learning low-dimensional data representations.

Demonstrates effectiveness across various application scenarios.

Abstract

Federated learning is a distributed machine learning mechanism where local devices collaboratively train a shared global model under the orchestration of a central server, while keeping all private data decentralized. In the system, model parameters and its updates are transmitted instead of raw data, and thus the communication bottleneck has become a key challenge. Besides, recent larger and deeper machine learning models also pose more difficulties in deploying them in a federated environment. In this paper, we design a federated two-stage learning framework that augments prototypical federated learning with a cut layer on devices and uses sign-based stochastic gradient descent with the majority vote method on model updates. Cut layer on devices learns informative and low-dimension representations of raw data locally, which helps reduce global model parameters and prevents data leakage. Sign-based SGD with the majority vote method for model updates also helps alleviate communication limitations. Empirically, we show that our system is an efficient and privacy preserving federated learning scheme and suits for general application scenarios.