Tighter Privacy Auditing of DP-SGD in the Hidden State Threat Model

TL;DR

This paper introduces a novel privacy auditing method for DP-SGD under a hidden state threat model, revealing that existing privacy bounds can be tightened by adversarial gradient crafting, especially when intermediate updates are concealed.

Contribution

It proposes an adversarial gradient crafting approach to more accurately audit privacy loss in DP-SGD under the hidden state threat model, challenging existing bounds.

Findings

Adversarial gradient sequences outperform previous auditing methods.

Concealing intermediate updates does not always improve privacy guarantees.

Existing privacy bounds can be tightened in certain regimes.

Abstract

Machine learning models can be trained with formal privacy guarantees via differentially private optimizers such as DP-SGD. In this work, we focus on a threat model where the adversary has access only to the final model, with no visibility into intermediate updates. In the literature, this hidden state threat model exhibits a significant gap between the lower bound from empirical privacy auditing and the theoretical upper bound provided by privacy accounting. To challenge this gap, we propose to audit this threat model with adversaries that craft a gradient sequence designed to maximize the privacy loss of the final model without relying on intermediate updates. Our experiments show that this approach consistently outperforms previous attempts at auditing the hidden state model. Furthermore, our results advance the understanding of achievable privacy guarantees within this threat model. Specifically, when the crafted gradient is inserted at every optimization step, we show that concealing the intermediate model updates in DP-SGD does not enhance the privacy guarantees. The situation is more complex when the crafted gradient is not inserted at every step: our auditing lower bound matches the privacy upper bound only for an adversarially-chosen loss landscape and a sufficiently large batch size. This suggests that existing privacy upper bounds can be improved in certain regimes.