R\'enyi Differential Privacy for Heavy-Tailed SDEs via Fractional Poincar\'e Inequalities

TL;DR

This paper develops Re9nyi differential privacy guarantees for heavy-tailed stochastic differential equations using fractional Poincare9 inequalities, addressing limitations of previous bounds that depended heavily on the number of parameters.

Contribution

It introduces the first Re9nyi DP guarantees for heavy-tailed SDEs and their discretizations, leveraging fractional Poincare9 inequalities to reduce dimension dependence.

Findings

Derived RDP guarantees for heavy-tailed SDEs.

Reduced dependence on the number of parameters.

Applicable to discretized algorithms with heavy-tailed noise.

Abstract

Characterizing the differential privacy (DP) of learning algorithms has become a major challenge in recent years. In parallel, many studies suggested investigating the behavior of stochastic gradient descent (SGD) with heavy-tailed noise, both as a model for modern deep learning models and to improve their performance. However, most DP bounds focus on light-tailed noise, where satisfactory guarantees have been obtained but the proposed techniques do not directly extend to the heavy-tailed setting. Recently, the first DP guarantees for heavy-tailed SGD were obtained. These results provide $(0,\delta)$-DP guarantees without requiring gradient clipping. Despite casting new light on the link between DP and heavy-tailed algorithms, these results have a strong dependence on the number of parameters and cannot be extended to other DP notions like the well-established R\'enyi differential privacy (RDP). In this work, we propose to address these limitations by deriving the first RDP guarantees for heavy-tailed SDEs, as well as their discretized counterparts. Our framework is based on new R\'enyi flow computations and the use of well-established fractional Poincar\'e inequalities. Under the assumption that such inequalities are satisfied, we obtain DP guarantees that have a much weaker dependence on the dimension compared to prior art.