Learning in a Large Function Space: Privacy-Preserving Mechanisms for SVM Learning

TL;DR

This paper introduces privacy-preserving mechanisms for SVM learning that balance differential privacy with utility, applicable to both finite and infinite-dimensional feature spaces, and provides theoretical bounds on privacy-utility trade-offs.

Contribution

The paper proposes two efficient privacy-preserving mechanisms for SVMs, including one for infinite-dimensional kernels, and establishes bounds on the achievable privacy-utility trade-off.

Findings

Mechanisms achieve high-probability utility close to non-private SVMs.

Proposed methods work for translation-invariant kernels in infinite-dimensional spaces.

Lower bounds show limitations on simultaneous privacy and utility guarantees.

Abstract

Several recent studies in privacy-preserving learning have considered the trade-off between utility or risk and the level of differential privacy guaranteed by mechanisms for statistical query processing. In this paper we study this trade-off in private Support Vector Machine (SVM) learning. We present two efficient mechanisms, one for the case of finite-dimensional feature mappings and one for potentially infinite-dimensional feature mappings with translation-invariant kernels. For the case of translation-invariant kernels, the proposed mechanism minimizes regularized empirical risk in a random Reproducing Kernel Hilbert Space whose kernel uniformly approximates the desired kernel with high probability. This technique, borrowed from large-scale learning, allows the mechanism to respond with a finite encoding of the classifier, even when the function class is of infinite VC dimension. Differential privacy is established using a proof technique from algorithmic stability. Utility--the mechanism's response function is pointwise epsilon-close to non-private SVM with probability 1-delta--is proven by appealing to the smoothness of regularized empirical risk minimization with respect to small perturbations to the feature mapping. We conclude with a lower bound on the optimal differential privacy of the SVM. This negative result states that for any delta, no mechanism can be simultaneously (epsilon,delta)-useful and beta-differentially private for small epsilon and small beta.