Training Large-Vocabulary Neural Language Models by Private Federated Learning for Resource-Constrained Devices

TL;DR

This paper introduces a novel approach combining Partial Embedding Updates, LoRA, and NCE to enable privacy-preserving training of large neural language models on resource-limited devices using federated learning.

Contribution

It proposes a new method that reduces privacy noise and memory demands, allowing large-vocabulary models to be trained efficiently on constrained devices.

Findings

Successful training of large-vocabulary models with privacy guarantees

Reduced memory usage enabling deployment on resource-constrained devices

Maintained model accuracy despite privacy-preserving modifications

Abstract

Federated Learning (FL) is a technique to train models using data distributed across devices. Differential Privacy (DP) provides a formal privacy guarantee for sensitive data. Our goal is to train a large neural network language model (NNLM) on compute-constrained devices while preserving privacy using FL and DP. However, the DP-noise introduced to the model increases as the model size grows, which often prevents convergence. We propose Partial Embedding Updates (PEU), a novel technique to decrease noise by decreasing payload size. Furthermore, we adopt Low Rank Adaptation (LoRA) and Noise Contrastive Estimation (NCE) to reduce the memory demands of large models on compute-constrained devices. This combination of techniques makes it possible to train large-vocabulary language models while preserving accuracy and privacy.