On the Robustness of Distributed Machine Learning against Transfer Attacks

TL;DR

This paper investigates the robustness of fully heterogeneous distributed machine learning models against transfer attacks, demonstrating significant improvements in robustness and accuracy tradeoffs through extensive experiments.

Contribution

It is the first to analyze the combined robustness of distributed ML during training and inference, showing enhanced resistance to transfer-based attacks.

Findings

Up to 40% increase in robust accuracy against transfer attacks on CIFAR10.

Distributed ML models outperform ensemble and federated learning in robustness.

Minimal impact on clean accuracy while improving robustness.

Abstract

Although distributed machine learning (distributed ML) is gaining considerable attention in the community, prior works have independently looked at instances of distributed ML in either the training or the inference phase. No prior work has examined the combined robustness stemming from distributing both the learning and the inference process. In this work, we explore, for the first time, the robustness of distributed ML models that are fully heterogeneous in training data, architecture, scheduler, optimizer, and other model parameters. Supported by theory and extensive experimental validation using CIFAR10 and FashionMNIST, we show that such properly distributed ML instantiations achieve across-the-board improvements in accuracy-robustness tradeoffs against state-of-the-art transfer-based attacks that could otherwise not be realized by current ensemble or federated learning instantiations. For instance, our experiments on CIFAR10 show that for the Common Weakness attack, one of the most powerful state-of-the-art transfer-based attacks, our method improves robust accuracy by up to 40%, with a minimal impact on clean task accuracy.