Lossless Anti-Distillation Sampling

TL;DR

LADS is a novel sampling scheme that prevents multi-account distillation attacks on generative models by ensuring responses are statistically identical for benign users while degrading the quality of distillation data.

Contribution

LADS introduces a privacy-preserving sampling method that maintains response quality for users and reduces distillation effectiveness by correlating harvested data.

Findings

LADS significantly reduces the performance of distilled models across tasks.

LADS maintains identical responses for individual users, ensuring no distortion.

Theoretical analysis shows LADS degrades distiller convergence rates.

Abstract

Frontier commercial generative models face a growing threat from distillation, whereby a distiller harvests generated responses and trains a competing model of its own at drastically lower cost. Existing defenses either rely on modifying the models outputs, thereby sacrificing response quality for benign users, or on behavioral detection methods, which can be readily circumvented by distributing queries across multiple accounts. In this work, we propose Lossless Anti-Distillation Sampling (LADS), a novel sampling scheme specifically designed to counter multi-account distillation while maintaining a lossless experience for benign users. Concretely, LADS derives the randomness underlying each generation from a private seed determined by the semantic content of the query and the number of times the user has queried the model. By construction, every benign user receives a response independently sampled from the original model at each visit, and thus experiences no distortion. In contrast, for a distiller, different accounts share latent randomness whenever their queries fall in the same semantic bucket. As a result, the harvested data becomes correlated, potentially reducing sample diversity and degrading generalization. Using uniform convergence theory, we show that LADS provably degrades the convergence rate of the distillers generalization gap relative to standard i.i.d. sampling in both unconditional and conditional generation settings. Experiments on image generation, mathematical reasoning, and code generation confirm that LADS substantially degrades the performance of distilled students while preserving exact statistical fidelity for individual users.