A Collocation-based Method for Addressing Challenges in Word-level Metric Differential Privacy

TL;DR

This paper introduces a collocation-based approach to improve semantic coherence and flexibility in word-level metric differential privacy for NLP, addressing limitations of existing methods by perturbing n-grams instead of individual words.

Contribution

The authors propose a novel collocation-based method that operates between word and sentence levels, enhancing semantic coherence and output variability in differentially private NLP applications.

Findings

Improved semantic coherence in privatized text outputs.

Enhanced flexibility with variable length outputs.

Effective privacy preservation demonstrated through utility and privacy tests.

Abstract

Applications of Differential Privacy (DP) in NLP must distinguish between the syntactic level on which a proposed mechanism operates, often taking the form of $\textit{word-level}$ or $\textit{document-level}$ privatization. Recently, several word-level $\textit{Metric}$ Differential Privacy approaches have been proposed, which rely on this generalized DP notion for operating in word embedding spaces. These approaches, however, often fail to produce semantically coherent textual outputs, and their application at the sentence- or document-level is only possible by a basic composition of word perturbations. In this work, we strive to address these challenges by operating $\textit{between}$ the word and sentence levels, namely with $\textit{collocations}$. By perturbing n-grams rather than single words, we devise a method where composed privatized outputs have higher semantic coherence and variable length. This is accomplished by constructing an embedding model based on frequently occurring word groups, in which unigram words co-exist with bi- and trigram collocations. We evaluate our method in utility and privacy tests, which make a clear case for tokenization strategies beyond the word level.