Differentially Private Model Merging

TL;DR

This paper introduces post-processing methods to merge existing models trained with different differential privacy levels into a single model meeting any specified privacy requirement without retraining.

Contribution

It proposes two novel post-processing techniques for privacy-preserving model merging and provides theoretical privacy analysis and empirical validation.

Findings

Effective privacy accounting for the proposed methods.

Successful empirical validation on synthetic and real datasets.

Clear characterization of privacy/utility tradeoffs.

Abstract

In machine learning, privacy requirements at inference or deployment time often evolve due to changing policies, regulations, or user preferences. In this work, we aim to construct a magnitude of models to satisfy any target differential privacy (DP) requirement without additional training, given a set of existing models trained on the same dataset with different privacy/utility tradeoffs. We propose two post-processing techniques, namely random selection and linear combination, to generate final private models satisfying any target privacy parameter. We provide privacy accounting of these approaches from the lens of R'enyi DP and privacy loss distributions on general problems, as well as on private mean estimation, where we precisely characterize the privacy/utility tradeoffs and compare the two mechanisms. Empirically, we demonstrate the effectiveness of our approaches and validate our analyses on several models and both synthetic and real-world datasets.