ImprovDML: Improved Trade-off in Private Byzantine-Resilient Distributed Machine Learning

TL;DR

ImprovDML introduces a decentralized framework for distributed machine learning that balances privacy, Byzantine resilience, and high accuracy through novel algorithms and privacy analysis.

Contribution

It proposes a new decentralized DML framework with resilient vector consensus and Gaussian noise for privacy, offering tighter convergence bounds and better privacy-accuracy trade-offs.

Findings

Achieves high model accuracy with privacy and Byzantine resilience

Provides convergence guarantees with tighter error bounds

Demonstrates improved privacy-accuracy trade-off through concentrated geo-privacy

Abstract

Jointly addressing Byzantine attacks and privacy leakage in distributed machine learning (DML) has become an important issue. A common strategy involves integrating Byzantine-resilient aggregation rules with differential privacy mechanisms. However, the incorporation of these techniques often results in a significant degradation in model accuracy. To address this issue, we propose a decentralized DML framework, named ImprovDML, that achieves high model accuracy while simultaneously ensuring privacy preservation and resilience to Byzantine attacks. The framework leverages a kind of resilient vector consensus algorithms that can compute a point within the normal (non-Byzantine) agents' convex hull for resilient aggregation at each iteration. Then, multivariate Gaussian noises are introduced to the gradients for privacy preservation. We provide convergence guarantees and derive asymptotic learning error bounds under non-convex settings, which are tighter than those reported in existing works. For the privacy analysis, we adopt the notion of concentrated geo-privacy, which quantifies privacy preservation based on the Euclidean distance between inputs. We demonstrate that it enables an improved trade-off between privacy preservation and model accuracy compared to differential privacy. Finally, numerical simulations validate our theoretical results.