ADDS: Adaptive Differentiable Sampling for Robust Multi-Party Learning

TL;DR

This paper introduces ADDS, a novel adaptive differentiable sampling framework that enhances multi-party learning by enabling clients to update customized subnets, reducing costs and improving robustness and convergence of the central model.

Contribution

It proposes a differentiable sampling strategy allowing clients to extract optimal local architectures, with minimal modifications to existing multi-party learning systems.

Findings

Reduces local computation and communication costs.

Speeds up convergence of the central model.

Improves robustness of multi-party learning.

Abstract

Distributed multi-party learning provides an effective approach for training a joint model with scattered data under legal and practical constraints. However, due to the quagmire of a skewed distribution of data labels across participants and the computation bottleneck of local devices, how to build smaller customized models for clients in various scenarios while providing updates appliable to the central model remains a challenge. In this paper, we propose a novel adaptive differentiable sampling framework (ADDS) for robust and communication-efficient multi-party learning. Inspired by the idea of dropout in neural networks, we introduce a network sampling strategy in the multi-party setting, which distributes different subnets of the central model to clients for updating, and the differentiable sampling rates allow each client to extract optimal local architecture from the supernet according to its private data distribution. The approach requires minimal modifications to the existing multi-party learning structure, and it is capable of integrating local updates of all subnets into the supernet, improving the robustness of the central model. The proposed framework significantly reduces local computation and communication costs while speeding up the central model convergence, as we demonstrated through experiments on real-world datasets.