PRIVET: Privacy Metric Based on Extreme Value Theory
Antoine Szatkownik (TAU, BioInfo), Aur\'elien Decelle, Beatriz Seoane (TAU), Nicolas Bereux (TAU), L\'eo Planche (BioInfo), Guillaume Charpiat (TAU), Burak Yelmen, Flora Jay (BioInfo, TAU), Cyril Furtlehner (TAU)
TL;DR
PRIVET is a novel privacy evaluation method using extreme value theory to assign individual privacy leak scores to synthetic data samples, improving detection of privacy risks across diverse data types.
Contribution
It introduces a sample-based, modality-agnostic algorithm for privacy assessment that outperforms existing global criteria methods.
Findings
Reliably detects memorization and privacy leakage in high-dimensional data.
Provides both dataset-level and sample-level privacy assessments.
Effective even with limited data or underfitting regimes.
Abstract
Deep generative models are often trained on sensitive data, such as genetic sequences, health data, or more broadly, any copyrighted, licensed or protected content. This raises critical concerns around privacy-preserving synthetic data, and more specifically around privacy leakage, an issue closely tied to overfitting. Existing methods almost exclusively rely on global criteria to estimate the risk of privacy failure associated to a model, offering only quantitative non interpretable insights. The absence of rigorous evaluation methods for data privacy at the sample-level may hinder the practical deployment of synthetic data in real-world applications. Using extreme value statistics on nearest-neighbor distances, we propose PRIVET, a generic sample-based, modality-agnostic algorithm that assigns an individual privacy leak score to each synthetic sample. We empirically demonstrate that…
Peer Reviews
Decision·Submitted to ICLR 2026
Another metric that addresses an important privacy problem. Authors provide some interesting examples based on pseudo-synthetic data that suits their distance metrics and embeddings.
Only a single experiment using an RBM-generated synthetic dataset is presented, which limits the generality of the findings. Including additional datasets would strengthen the empirical validation. Moreover, the current results do not clearly demonstrate whether the proposed approach detects any actual privacy risk. One potential improvement would be to employ a data generator with a controlled privacy parameter—for example, a differentially private generator with varying epsilon values—and ev
1) Broad empirical coverage. The study spans high-dimensional genetics, computer vision with modern embeddings, and a membership-attack scenario, which together demonstrate versatility across modalities and training regimes, including underfitting. 2) Interpretability and actionable outputs. The ability to flag specific synthetic samples as probable leaks and to summarize with an estimated number of leaks is useful in governance workflows. The method surfaces when embedding choices compromise de
1) Embedding dependence undermines "modality-agnostic" positioning. Results show that detectability varies substantially with the representation, with failures for certain image transforms under DINOv2 and improvements with wavelets. The method is only as good as the embedding and distance, which weakens the claim of generality. A guidance section on selecting or learning privacy-aware embeddings, or an adaptive metric learning step, would be valuable. 2) Figure~1 readability. The numerical anno
1. The paper applies extreme value theory to model the nearest-neighbor distance tails, providing a statistically grounded method for privacy leakage detection. 2. PRIVET can generate interpretable, per-sample privacy scores while remaining consistent with dataset-level statistics, which bridges the gap between global metrics (like AATS, FLD) and local boolean methods (like AUTH). 3. Experiments on both genetic and image datasets demonstrate the versatility of the method, showing good scalabil
1. Table 1’s distinction between “interpretable” and “non-interpretable” metrics is not clearly defined. It appears that *interpretability* is equated with having a boolean “leak / not leak” label, but the table also lists a “real value” column without clear justification. Moreover, the rationale for classifying methods such as AUTH as interpretable only at the sample level, but not at the dataset level, is insufficiently explained. 2. The paper claims that prior metrics like FLD and PQMass fai
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsPrivacy-Preserving Technologies in Data · Adversarial Robustness in Machine Learning · Advanced Malware Detection Techniques