LDPKiT: Superimposing Remote Queries for Privacy-Preserving Local Model Training

TL;DR

LDPKiT is a privacy-preserving framework for model extraction that uses superimposition of in-distribution samples under local differential privacy, achieving high utility with strong privacy guarantees.

Contribution

It introduces a novel superimposition technique for generating in-distribution samples to enable effective knowledge transfer under local differential privacy.

Findings

LDPKiT maintains high accuracy at strong privacy levels.

Performance improves with larger datasets.

Theoretical analysis explains accuracy gains.

Abstract

Users of modern Machine Learning (ML) cloud services face a privacy conundrum -- on one hand, they may have concerns about sending private data to the service for inference, but on the other hand, for specialized models, there may be no alternative but to use the proprietary model of the ML service. In this work, we present LDPKiT, a framework for non-adversarial, privacy-preserving model extraction that leverages a user's private in-distribution data while bounding privacy leakage. LDPKiT introduces a novel superimposition technique that generates approximately in-distribution samples, enabling effective knowledge transfer under local differential privacy (LDP). Experiments on Fashion-MNIST, SVHN, and PathMNIST demonstrate that LDPKiT consistently improves utility while maintaining privacy, with benefits that become more pronounced at stronger noise levels. For example, on SVHN, LDPKiT achieves nearly the same inference accuracy at $\epsilon=1.25$ as at $\epsilon=2.0$, yielding stronger privacy guarantees with less than a 2% accuracy reduction. We further conduct sensitivity analyses to examine the effect of dataset size on performance and provide a systematic analysis of latent space representations, offering theoretical insights into the accuracy gains of LDPKiT.