UniAud: A Unified Auditing Framework for High Auditing Power and Utility with One Training Run

TL;DR

UniAud introduces a unified, efficient auditing framework that enhances privacy verification in differentially private models, achieving high auditing power with minimal utility loss and significantly improved efficiency over traditional methods.

Contribution

The paper presents UniAud, a novel unified framework for data-independent and data-dependent DP auditing, utilizing uncorrelated canaries and multi-task learning to maximize auditing power and efficiency.

Findings

UniAud matches state-of-the-art auditing results with only one training run.

It achieves the best efficiency-utility trade-off across vision and language tasks.

Provides meaningful auditing with minimal utility degradation.

Abstract

Differentially private (DP) optimization has been widely adopted as a standard approach to provide rigorous privacy guarantees for training datasets. DP auditing verifies whether a model trained with DP optimization satisfies its claimed privacy level by estimating empirical privacy lower bounds through hypothesis testing. Recent O(1) frameworks improve auditing efficiency by checking the membership status of multiple audit samples in a single run, rather than checking individual samples across multiple runs. However, we reveal that there is no free lunch for this improved efficiency: data dependency and an implicit conflict between auditing and utility impair the tightness of the auditing results. Addressing these challenges, our key insights include reducing data dependency through uncorrelated data and resolving the auditing-utility conflict by decoupling the criteria for effective auditing and separating objectives for utility and auditing. We first propose a unified framework, UniAud, for data-independent auditing that maximizes auditing power through a novel uncorrelated canary construction and a self-comparison framework. We then extend this framework as UniAud++ for data-dependent auditing, optimizing the auditing and utility trade-off through multi-task learning with separate objectives for auditing and training. Experimental results validate that our black-box O(1) framework matches the state-of-the-art auditing results of O(T) auditing with thousands of runs, demonstrating the best efficiency-auditing trade-off across vision and language tasks. Additionally, our framework provides meaningful auditing with only slight utility degradation compared to standard DP training, showing the optimal utility-auditing trade-off and the benefit of requiring no extra training for auditing.