Federated Boosted Decision Trees with Differential Privacy

TL;DR

This paper introduces a federated learning framework for gradient boosted decision trees that incorporates differential privacy, balancing high utility with strong privacy guarantees in distributed data settings.

Contribution

It presents a novel framework for differentially private federated GBDT models, extending existing methods and optimizing privacy-utility trade-offs.

Findings

Achieves high utility while maintaining strong privacy guarantees.

Extends existing federated decision tree approaches with differential privacy.

Provides a flexible framework adaptable to various hyperparameters.

Abstract

There is great demand for scalable, secure, and efficient privacy-preserving machine learning models that can be trained over distributed data. While deep learning models typically achieve the best results in a centralized non-secure setting, different models can excel when privacy and communication constraints are imposed. Instead, tree-based approaches such as XGBoost have attracted much attention for their high performance and ease of use; in particular, they often achieve state-of-the-art results on tabular data. Consequently, several recent works have focused on translating Gradient Boosted Decision Tree (GBDT) models like XGBoost into federated settings, via cryptographic mechanisms such as Homomorphic Encryption (HE) and Secure Multi-Party Computation (MPC). However, these do not always provide formal privacy guarantees, or consider the full range of hyperparameters and implementation settings. In this work, we implement the GBDT model under Differential Privacy (DP). We propose a general framework that captures and extends existing approaches for differentially private decision trees. Our framework of methods is tailored to the federated setting, and we show that with a careful choice of techniques it is possible to achieve very high utility while maintaining strong levels of privacy.