Multi-Party Dual Learning

TL;DR

This paper introduces a multi-party dual learning framework that enhances model training across distributed parties with limited or poor-quality data, while ensuring privacy and achieving superior performance over existing methods.

Contribution

The paper proposes a novel multi-party dual learning framework that leverages dual task relationships and introduces differential privacy to improve distributed learning with limited data.

Findings

Significant performance improvement over state-of-the-art methods

Effective privacy preservation with feature-oriented differential privacy

Minimal modifications needed for existing multi-party learning structures

Abstract

The performance of machine learning algorithms heavily relies on the availability of a large amount of training data. However, in reality, data usually reside in distributed parties such as different institutions and may not be directly gathered and integrated due to various data policy constraints. As a result, some parties may suffer from insufficient data available for training machine learning models. In this paper, we propose a multi-party dual learning (MPDL) framework to alleviate the problem of limited data with poor quality in an isolated party. Since the knowledge sharing processes for multiple parties always emerge in dual forms, we show that dual learning is naturally suitable to handle the challenge of missing data, and explicitly exploits the probabilistic correlation and structural relationship between dual tasks to regularize the training process. We introduce a feature-oriented differential privacy with mathematical proof, in order to avoid possible privacy leakage of raw features in the dual inference process. The approach requires minimal modifications to the existing multi-party learning structure, and each party can build flexible and powerful models separately, whose accuracy is no less than non-distributed self-learning approaches. The MPDL framework achieves significant improvement compared with state-of-the-art multi-party learning methods, as we demonstrated through simulations on real-world datasets.