A Neighbourhood-Aware Differential Privacy Mechanism for Static Word Embeddings

TL;DR

This paper introduces a neighbourhood-aware differential privacy mechanism for static word embeddings that adaptively adds noise based on local neighborhood structure, improving privacy and utility in NLP tasks.

Contribution

The paper presents a novel NADP mechanism that considers word neighborhoods in embedding space to optimize privacy noise application, outperforming existing DP methods.

Findings

NADP outperforms Laplacian, Gaussian, and Mahalanobis DP mechanisms in downstream tasks.

NADP guarantees higher privacy levels while maintaining utility.

Constructing a neighborhood graph enables adaptive noise application.

Abstract

We propose a Neighbourhood-Aware Differential Privacy (NADP) mechanism considering the neighbourhood of a word in a pretrained static word embedding space to determine the minimal amount of noise required to guarantee a specified privacy level. We first construct a nearest neighbour graph over the words using their embeddings, and factorise it into a set of connected components (i.e. neighbourhoods). We then separately apply different levels of Gaussian noise to the words in each neighbourhood, determined by the set of words in that neighbourhood. Experiments show that our proposed NADP mechanism consistently outperforms multiple previously proposed DP mechanisms such as Laplacian, Gaussian, and Mahalanobis in multiple downstream tasks, while guaranteeing higher levels of privacy.