An exponential mechanism based on quadratic approximations for fine-tuning machine learning models with privacy guarantees

TL;DR

This paper introduces a differentially private fine-tuning method for machine learning models using an exponential mechanism based on quadratic approximations, ensuring privacy while maintaining accuracy.

Contribution

It develops a novel exponential mechanism leveraging quadratic approximations for privacy-preserving fine-tuning, with scalable random projection strategies and theoretical guarantees.

Findings

Achieves competitive accuracy on MNIST and MIMIC datasets.

Provides exact sampling from a multivariate normal distribution.

Ensures differential privacy with theoretical guarantees.

Abstract

Fine-tuning adapts a pretrained machine learning model to a small, sensitive dataset, but this process risks memorizing individual new data points, making the model vulnerable to adversaries who seek to extract sensitive information. In this work, we develop a randomized algorithm based on the exponential mechanism for fine-tuning while ensuring differential privacy. Our key idea is to construct a simple utility function that combines a local quadratic approximation of the pretrained model with information from the new dataset. The resulting exponential mechanism admits exact sampling from a multivariate normal distribution in closed form. We establish theoretical privacy guarantees, sensitivity bounds, and accuracy estimations for our method. We further introduce a random-projection strategy that makes the approach scalable to high-dimensional models. Numerical experiments on the MNIST benchmark and the MIMIC clinical dataset demonstrate competitive performance against existing differentially private fine-tuning techniques.