DP-MLM: Differentially Private Text Rewriting Using Masked Language Models

TL;DR

This paper introduces DP-MLM, a novel differentially private text rewriting method using masked language models, which improves privacy-utility trade-offs and offers greater customization over previous autoregressive approaches.

Contribution

The paper proposes DP-MLM, a new privacy-preserving text rewriting technique leveraging masked language models for better utility and flexibility.

Findings

MLMs outperform autoregressive models at low privacy levels

DP-MLM provides improved utility preservation

The method allows greater customization of rewriting mechanisms

Abstract

The task of text privatization using Differential Privacy has recently taken the form of $\textit{text rewriting}$, in which an input text is obfuscated via the use of generative (large) language models. While these methods have shown promising results in the ability to preserve privacy, these methods rely on autoregressive models which lack a mechanism to contextualize the private rewriting process. In response to this, we propose $\textbf{DP-MLM}$, a new method for differentially private text rewriting based on leveraging masked language models (MLMs) to rewrite text in a semantically similar $\textit{and}$ obfuscated manner. We accomplish this with a simple contextualization technique, whereby we rewrite a text one token at a time. We find that utilizing encoder-only MLMs provides better utility preservation at lower $\varepsilon$ levels, as compared to previous methods relying on larger models with a decoder. In addition, MLMs allow for greater customization of the rewriting mechanism, as opposed to generative approaches. We make the code for $\textbf{DP-MLM}$ public and reusable, found at https://github.com/sjmeis/DPMLM .