Matrix Sketching for Secure Collaborative Machine Learning

TL;DR

This paper introduces Double-Blind Collaborative Learning (DBCL), a practical matrix sketching method that enhances privacy in federated learning without compromising accuracy or efficiency.

Contribution

The paper proposes DBCL, a novel matrix sketching technique that prevents gradient-based privacy attacks in collaborative learning while maintaining model performance.

Findings

DBCL effectively prevents gradient-based privacy inferences.

DBCL does not significantly increase computation or communication costs.

DBCL maintains test accuracy comparable to standard methods.

Abstract

Collaborative learning allows participants to jointly train a model without data sharing. To update the model parameters, the central server broadcasts model parameters to the clients, and the clients send updating directions such as gradients to the server. While data do not leave a client device, the communicated gradients and parameters will leak a client's privacy. Attacks that infer clients' privacy from gradients and parameters have been developed by prior work. Simple defenses such as dropout and differential privacy either fail to defend the attacks or seriously hurt test accuracy. We propose a practical defense which we call Double-Blind Collaborative Learning (DBCL). The high-level idea is to apply random matrix sketching to the parameters (aka weights) and re-generate random sketching after each iteration. DBCL prevents clients from conducting gradient-based privacy inferences which are the most effective attacks. DBCL works because from the attacker's perspective, sketching is effectively random noise that outweighs the signal. Notably, DBCL does not much increase computation and communication costs and does not hurt test accuracy at all.