Nonparametric Variational Differential Privacy via Embedding Parameter Clipping

TL;DR

This paper introduces a parameter clipping method for nonparametric variational differential privacy models, improving privacy guarantees and utility by constraining latent representations based on Rénnyi Divergence bounds.

Contribution

It proposes a theoretically grounded parameter clipping strategy derived from RD bounds to enhance privacy and utility in NVIB-based models.

Findings

Clipped models achieve tighter Rénnyi Divergence bounds.

Clipping improves privacy guarantees.

Clipped models perform better on downstream tasks.

Abstract

The nonparametric variational information bottleneck (NVIB) provides the foundation for nonparametric variational differential privacy (NVDP), a framework for building privacy-preserving language models. However, the learned latent representations can drift into regions with high information content, leading to poor privacy guarantees, but also low utility due to numerical instability during training. In this work, we introduce a principled parameter clipping strategy to directly address this issue. Our method is mathematically derived from the objective of minimizing the R\'enyi Divergence (RD) upper bound, yielding specific, theoretically grounded constraints on the posterior mean, variance, and mixture weight parameters. We apply our technique to an NVIB based model and empirically compare it against an unconstrained baseline. Our findings demonstrate that the clipped model consistently achieves tighter RD bounds, implying stronger privacy, while simultaneously attaining higher performance on several downstream tasks. This work presents a simple yet effective method for improving the privacy-utility trade-off in variational models, making them more robust and practical.