Communication-Learning Co-Design for Differentially Private Over-the-Air Federated Distillation

TL;DR

This paper introduces a differentially private over-the-air federated distillation framework that leverages wireless channel superposition to enhance communication efficiency and privacy in large-scale federated learning.

Contribution

It proposes a novel DP over-the-air FD method with a co-design approach to optimize convergence and privacy, including analytical derivations and closed-form solutions.

Findings

Achieves better learning-privacy trade-off than traditional FL

Reduces communication overhead significantly

Provides analytical convergence and privacy loss analysis

Abstract

The ever-growing learning model size nowadays challenges the communication efficiency and privacy preservation of the traditional federated learning (FL). In this paper, we propose a novel differentially private (DP) over-the-air federated distillation (FD) framework, where wireless devices (WDs) periodically share noise-perturbed model outputs with the parameter server by harnessing the superposition property of multi-access channels. Accordingly, over-the-air FD enables the shared responsibility of the DP preservation on the low-dimensional disclosed signals among WDs. We study the communication-learning co-design problem in differentially private over-the-air FD, aiming to maximize the learning convergence rate while meeting the transmit power and DP requirements of WDs. The main challenge is rooted in the intractable learning and privacy analysis in over-the-air FD, together with the strong coupling among the decision variables spanning two timescales. To tackle this problem, we first derive the analytical learning convergence rate and privacy losses of WDs, based on which the optimal transceiver design per FD round and long-term training rounds decision are obtained in the closed forms. Numerical results demonstrate that the proposed differentially private over-the-air FD approach achieves a better learning-privacy trade-off with largely-reduced communication overhead than the conventional FL benchmarks.