One size does not fit all: Investigating strategies for differentially-private learning across NLP tasks

TL;DR

This paper analyzes various privacy-preserving strategies in NLP across multiple tasks and models, revealing that no single approach works best universally and each task requires tailored privacy solutions.

Contribution

It provides an extensive empirical evaluation of privacy strategies in NLP, highlighting the need for task-specific privacy approaches rather than one-size-fits-all solutions.

Findings

Privacy strategies vary in effectiveness across tasks

No universal privacy approach outperforms others

Task-specific privacy tuning is necessary for optimal performance

Abstract

Preserving privacy in contemporary NLP models allows us to work with sensitive data, but unfortunately comes at a price. We know that stricter privacy guarantees in differentially-private stochastic gradient descent (DP-SGD) generally degrade model performance. However, previous research on the efficiency of DP-SGD in NLP is inconclusive or even counter-intuitive. In this short paper, we provide an extensive analysis of different privacy preserving strategies on seven downstream datasets in five different `typical' NLP tasks with varying complexity using modern neural models based on BERT and XtremeDistil architectures. We show that unlike standard non-private approaches to solving NLP tasks, where bigger is usually better, privacy-preserving strategies do not exhibit a winning pattern, and each task and privacy regime requires a special treatment to achieve adequate performance.