Vertical Federated Learning Hybrid Local Pre-training

TL;DR

This paper introduces VFLHLP, a novel approach in vertical federated learning that pre-trains local models to improve performance and address data scarcity issues, showing significant improvements on real-world datasets.

Contribution

The paper proposes a hybrid local pre-training method for VFL that leverages pre-trained local models to enhance federated learning performance, especially with unaligned data.

Findings

Achieves superior performance over baseline methods on real-world datasets.

Effectively utilizes unaligned data through pre-training to mitigate data scarcity.

Ablation study confirms the contribution of each component in VFLHLP.

Abstract

Vertical Federated Learning (VFL), which has a broad range of real-world applications, has received much attention in both academia and industry. Enterprises aspire to exploit more valuable features of the same users from diverse departments to boost their model prediction skills. VFL addresses this demand and concurrently secures individual parties from exposing their raw data. However, conventional VFL encounters a bottleneck as it only leverages aligned samples, whose size shrinks with more parties involved, resulting in data scarcity and the waste of unaligned data. To address this problem, we propose a novel VFL Hybrid Local Pre-training (VFLHLP) approach. VFLHLP first pre-trains local networks on the local data of participating parties. Then it utilizes these pre-trained networks to adjust the sub-model for the labeled party or enhance representation learning for other parties during downstream federated learning on aligned data, boosting the performance of federated models. The experimental results on real-world advertising datasets, demonstrate that our approach achieves the best performance over baseline methods by large margins. The ablation study further illustrates the contribution of each technique in VFLHLP to its overall performance.