Private Blind Model Averaging - Distributed, Non-interactive, and Convergent

TL;DR

This paper introduces BlindAvg, a non-interactive, differentially private model averaging method for distributed learning, demonstrating its convergence to centralized models under certain regularization and proposing a new private ERM learner with improved utility.

Contribution

It analyzes the convergence and utility of BlindAvg in convex, smooth ERM settings and proposes SoftmaxReg, a new private learner with better privacy-utility tradeoff.

Findings

BlindAvg converges to centralized models with strong L2-regularization.

SoftmaxReg outperforms SVM in privacy-utility tradeoff.

Empirical evaluation on CIFAR-10, CIFAR-100, and EMNIST datasets.

Abstract

Distributed differentially private learning techniques enable a large number of users to jointly learn a model without having to first centrally collect the training data. At the same time, neither the communication between the users nor the resulting model shall leak information about the training data. This kind of learning technique can be deployed to edge devices if it can be scaled up to a large number of users, particularly if the communication is reduced to a minimum: no interaction, i.e., each party only sends a single message. The best previously known methods are based on gradient averaging, which inherently requires many synchronization rounds. A promising non-interactive alternative to gradient averaging relies on so-called output perturbation: each user first locally finishes training and then submits its model for secure averaging without further synchronization. We analyze this paradigm, which we coin blind model averaging (BlindAvg), in the setting of convex and smooth empirical risk minimization (ERM) like a support vector machine (SVM). While the required noise scale is asymptotically the same as in the centralized setting, it is not well understood how close BlindAvg comes to centralized learning, i.e., its utility cost. We characterize and boost the privacy-utility tradeoff of BlindAvg with two contributions: First, we prove that BlindAvg converges towards the centralized setting for a sufficiently strong L2-regularization for a non-smooth SVM learner. Second, we introduce the novel differentially private convex and smooth ERM learner SoftmaxReg that has a better privacy-utility tradeoff than an SVM in a multi-class setting. We evaluate our findings on three datasets (CIFAR-10, CIFAR-100, and Federated EMNIST) and provide an ablation in an artificially extreme non-IID scenario.