When Gradient Clipping Becomes a Control Mechanism for Differential Privacy in Deep Learning

TL;DR

This paper introduces a novel control-driven gradient clipping method for differentially private deep learning that adaptively adjusts the clipping threshold using spectral diagnostics, improving privacy-utility trade-offs without extra privacy cost.

Contribution

A new adaptive clipping strategy using spectral diagnostics and feedback control that reduces computational overhead and sensitivity to dataset and architecture variations.

Findings

Improved privacy-utility trade-off in DP training.

Reduced computational overhead compared to existing methods.

Threshold adaptation based on spectral diagnostics enhances training stability.

Abstract

Privacy-preserving training on sensitive data commonly relies on differentially private stochastic optimization with gradient clipping and Gaussian noise. The clipping threshold is a critical control knob: if set too small, systematic over-clipping induces optimization bias; if too large, injected noise dominates updates and degrades accuracy. Existing adaptive clipping methods often depend on per-example gradient norm statistics, adding computational overhead and introducing sensitivity to datasets and architectures. We propose a control-driven clipping strategy that adapts the threshold using a lightweight, weight-only spectral diagnostic computed from model parameters. At periodic probe steps, the method analyzes a designated weight matrix via spectral decomposition and estimates a heavy-tailed spectral indicator associated with training stability. This indicator is smoothed over time and fed into a bounded feedback controller that updates the clipping threshold multiplicatively in the log domain. Because the controller uses only parameters produced during privacy-preserving training, the resulting threshold updates are post-processing and do not increase privacy loss beyond that of the underlying DP optimizer under standard composition accounting.