LLM-based Privacy Data Augmentation Guided by Knowledge Distillation with a Distribution Tutor for Medical Text Classification

TL;DR

This paper introduces a novel privacy-preserving data augmentation method for medical text classification using large language models, knowledge distillation, and a distribution tutor to generate pseudo samples with privacy guarantees.

Contribution

It proposes a DP-based data augmentation framework utilizing a knowledge distillation model and a distribution tutor to enhance privacy and data quality in private domain text classification.

Findings

The method effectively generates private pseudo samples with strong privacy guarantees.

Empirical results show improved classification performance with privacy protection.

Theoretical analysis confirms the privacy bounds of the proposed approach.

Abstract

As sufficient data are not always publically accessible for model training, researchers exploit limited data with advanced learning algorithms or expand the dataset via data augmentation (DA). Conducting DA in private domain requires private protection approaches (i.e. anonymization and perturbation), but those methods cannot provide protection guarantees. Differential privacy (DP) learning methods theoretically bound the protection but are not skilled at generating pseudo text samples with large models. In this paper, we transfer DP-based pseudo sample generation task to DP-based generated samples discrimination task, where we propose a DP-based DA method with a LLM and a DP-based discriminator for text classification on private domains. We construct a knowledge distillation model as the DP-based discriminator: teacher models, accessing private data, teaches students how to select private samples with calibrated noise to achieve DP. To constrain the distribution of DA's generation, we propose a DP-based tutor that models the noised private distribution and controls samples' generation with a low privacy cost. We theoretically analyze our model's privacy protection and empirically verify our model.