DP-LAC: Lightweight Adaptive Clipping for Differentially Private Federated Fine-tuning of Language Models

TL;DR

DP-LAC is a novel adaptive clipping method for differentially private federated learning of language models, which estimates and adjusts the clipping threshold efficiently without extra privacy cost or hyperparameter tuning.

Contribution

It introduces a private histogram-based initial threshold estimation and adaptive adjustment during training, improving privacy-utility trade-offs in federated learning.

Findings

DP-LAC outperforms state-of-the-art adaptive clipping methods.

Achieves an average accuracy gain of 6.6%.

Does not require additional privacy budget or hyperparameters.

Abstract

Federated learning (FL) enables the collaborative training of large-scale language models (LLMs) across edge devices while keeping user data on-device. However, FL still exposes sensitive information through client-provided gradients. Differentially private stochastic gradient descent (DP-SGD) mitigates this risk by clipping each client's contribution to a threshold $C$ and adding noise proportional to $C$. Existing adaptive clipping techniques dynamically adjust $C$ but demand tedious hyperparameter tuning, which can erode the privacy budget. In this paper, we introduce DP-LAC, a method that first estimates an initial clipping threshold within an order of magnitude of the optimum using private histogram estimation, and then adapts this threshold during training without consuming additional privacy budget or introducing new hyperparameters. Empirical results show that DP-LAC outperforms both state-of-the-art adaptive clipping methods and vanilla DP-SGD, achieving an average accuracy gain of $6.6\%$.