Mitigating Noise Detriment in Differentially Private Federated Learning with Model Pre-training

TL;DR

This paper introduces Pretrain-DPFL, a framework for fine-tuning pre-trained models in Differentially Private Federated Learning, which improves accuracy by systematically analyzing strategies and providing theoretical and empirical validation.

Contribution

It proposes a systematic evaluation of fine-tuning strategies in DPFL, with theoretical convergence analysis and extensive experiments demonstrating improved privacy-utility trade-offs.

Findings

Pretrain-DPFL achieves 25.22% higher accuracy than scratch training.

Unified-tuning outperforms other strategies in mitigating noise.

Theoretical conditions identify optimal fine-tuning strategies.

Abstract

Differentially Private Federated Learning (DPFL) strengthens privacy protection by perturbing model gradients with noise, though at the cost of reduced accuracy. Although prior empirical studies indicate that initializing from pre-trained rather than random parameters can alleviate noise disturbance, the problem of optimally fine-tuning pre-trained models in DPFL remains unaddressed. In this paper, we propose Pretrain-DPFL, a framework that systematically evaluates three most representative fine-tuning strategies: full-tuning (FT), head-tuning (HT), and unified-tuning(UT) combining HT followed by FT. Through convergence analysis under smooth non-convex loss, we establish theoretical conditions for identifying the optimal fine-tuning strategy in Pretrain-DPFL, thereby maximizing the benefits of pre-trained models in mitigating noise disturbance. Extensive experiments across multiple datasets demonstrate Pretrain-DPFL's superiority, achieving $25.22\%$ higher accuracy than scratch training and outperforming the second-best baseline by $8.19\%$, significantly improving the privacy-utility trade-off in DPFL.