LLM-CEG: Extending the Classification Error Gauge Framework for Privacy Auditing of Large Language Models

TL;DR

This paper introduces LLM-CEG, a framework for privacy auditing of large language models that balances privacy and utility using empirical measures and differential privacy tuning.

Contribution

It extends the CEG framework to LLMs, proposing a systematic privacy auditing method with a prototype demonstrating significant privacy and utility improvements.

Findings

DP-SGD reduces MIA attacker advantage by 71.5%.

DP-SGD improves out-of-distribution utility by 47-50%.

Differential privacy acts as implicit regularization in LLM fine-tuning.

Abstract

This paper extends the Classification Error Gauge (x-CEG) framework, originally developed for measuring the privacy-utility trade-off in tabular datasets, to privacy auditing of Large Language Models (LLMs). We propose LLM-CEG, a systematic framework that employs membership inference attack (MIA) success rates as an empirical privacy gauge and model perplexity as a utility gauge, iteratively adjusting differential privacy parameters until both thresholds are jointly satisfied. A proof-of-concept prototype fine-tunes DistilGPT-2 on a synthetic clinical PII dataset under four privacy regimes using DP-SGD. Results indicate that DP-SGD reduces MIA attacker advantage by 71.5% while simultaneously improving out-of-distribution utility by 47-50% relative to the overfitted baseline, suggesting that differential privacy may act as implicit regularization under narrow fine-tuning conditions. We further extend the SIED engineering framework to the LLM context as LLM-SIED, providing an auditable, regulator-aligned process for privacy-compliant LLM deployment.