Balance is key: Private median splits yield high-utility random trees

TL;DR

This paper introduces DiPriMe forests, a differentially private ensemble method that uses median-based splits to improve utility in private classification and regression tasks.

Contribution

It proposes a novel tree-splitting technique using private medians to enhance privacy-utility trade-offs in random forests.

Findings

High utility demonstrated both theoretically and empirically.

Balanced leaf nodes improve privacy and accuracy.

Avoiding low occupancy leaves reduces noise impact.

Abstract

Random forests are a popular method for classification and regression due to their versatility. However, this flexibility can come at the cost of user privacy, since training random forests requires multiple data queries, often on small, identifiable subsets of the training data. Privatizing these queries typically comes at a high utility cost, in large part because we are privatizing queries on small subsets of the data, which are easily corrupted by added noise. In this paper, we propose DiPriMe forests, a novel tree-based ensemble method for differentially private regression and classification, which is appropriate for real or categorical covariates. We generate splits using a differentially private version of the median, which encourages balanced leaf nodes. By avoiding low occupancy leaf nodes, we avoid high signal-to-noise ratios when privatizing the leaf node sufficient statistics. We show theoretically and empirically that the resulting algorithm exhibits high utility, while ensuring differential privacy.