TL;DR
This paper models the trade-off in distillation attacks as a minimax game, proposing adaptive evaluation and defense strategies that reveal the difficulty of preventing distillation while maintaining efficiency.
Contribution
It introduces a game-theoretic framework for understanding distillation attacks and defenses, including a practical PoE defense method and adaptive evaluation techniques.
Findings
Adaptive students recover more capability than passive evaluation suggests.
PoE defense narrows robustness gap while remaining cost-effective.
Strong distillation remains difficult to prevent against adaptive attacks.
Abstract
Distillation attacks create a deployment trade-off for model providers: the same outputs that make a model more useful can also make it easier to imitate. We study this trade-off through a minimax game between a utility-constrained teacher and an adaptive student. Our framework yields tractable one-sided response rules: an adaptive evaluation rule in which the student reweights high-value examples, and a teacher-side defense template that suppresses outputs most useful for distillation. From a cheap proxy for example value, we derive Product-of-Experts (PoE), a simple forward-pass-only defense that combines the teacher with a proxy student during generation. Empirically, adaptive evaluation reveals a large passive--adaptive gap: on state-of-the-art defenses, adaptive students recover substantially more capability than passive evaluation suggests on GSM8K and MATH. Under this stronger…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
