Efficient and Private: Memorisation under differentially private parameter-efficient fine-tuning in language models

TL;DR

This paper explores how parameter-efficient fine-tuning methods can effectively balance performance and privacy in large language models by reducing memorisation risks and computational costs under differential privacy constraints.

Contribution

It demonstrates that PEFT methods match full fine-tuning performance while enhancing privacy and efficiency, addressing limitations of traditional DP fine-tuning approaches.

Findings

PEFT achieves comparable accuracy to full fine-tuning.

PEFT significantly reduces privacy leakage and memorisation.

Data poisoning experiments show improved privacy protection with PEFT.

Abstract

Fine-tuning large language models (LLMs) for specific tasks introduces privacy risks, as models may inadvertently memorise and leak sensitive training data. While Differential Privacy (DP) offers a solution to mitigate these risks, it introduces significant computational and performance trade-offs, particularly with standard fine-tuning approaches. Previous work has primarily focused on full-parameter updates, which are computationally intensive and may not fully leverage DPs potential in large models. In this work, we address these shortcomings by investigating Parameter-Efficient Fine-Tuning (PEFT) methods under DP constraints. We show that PEFT methods achieve comparable performance to standard fine-tuning while requiring fewer parameters and significantly reducing privacy leakage. Furthermore, we incorporate a data poisoning experiment involving intentional mislabelling to assess model memorisation and directly measure privacy risks. Our findings indicate that PEFT methods not only provide a promising alternative but also serve as a complementary approach for privacy-preserving, resource-efficient fine-tuning of LLMs.