Differential Privacy, Linguistic Fairness, and Training Data Influence: Impossibility and Possibility Theorems for Multilingual Language Models

TL;DR

This paper explores the complex trade-offs between differential privacy, linguistic fairness, and transparency in multilingual language models, showing some objectives are compatible while others conflict, and emphasizes the need for joint optimization.

Contribution

It provides theoretical insights and experimental evidence on the compatibility and conflicts among privacy, fairness, and transparency objectives in multilingual models.

Findings

Multilingual compression and linguistic fairness can coexist with differential privacy.

Differential privacy conflicts with training data influence sparsity, affecting transparency.

Joint optimization is necessary to balance these competing objectives.

Abstract

Language models such as mBERT, XLM-R, and BLOOM aim to achieve multilingual generalization or compression to facilitate transfer to a large number of (potentially unseen) languages. However, these models should ideally also be private, linguistically fair, and transparent, by relating their predictions to training data. Can these requirements be simultaneously satisfied? We show that multilingual compression and linguistic fairness are compatible with differential privacy, but that differential privacy is at odds with training data influence sparsity, an objective for transparency. We further present a series of experiments on two common NLP tasks and evaluate multilingual compression and training data influence sparsity under different privacy guarantees, exploring these trade-offs in more detail. Our results suggest that we need to develop ways to jointly optimize for these objectives in order to find practical trade-offs.