DistillGuard: Evaluating Defenses Against LLM Knowledge Distillation

TL;DR

This paper introduces DistillGuard, a framework for evaluating output-level defenses against knowledge distillation from large language models, revealing their limited effectiveness across tasks.

Contribution

We propose a systematic evaluation framework and taxonomy for defenses, providing comprehensive insights into their strengths and limitations.

Findings

Most output defenses are ineffective against naive attackers

Chain-of-thought removal impairs mathematical reasoning significantly

Data poisoning mainly affects conversational fluency

Abstract

Knowledge distillation from proprietary LLM APIs poses a growing threat to model providers, yet defenses against this attack remain fragmented and unevaluated. We present DistillGuard, a framework for systematically evaluating output-level defenses against LLM knowledge distillation. We introduce a taxonomy of three defense categories -- output perturbation, data poisoning, and information throttling -- and evaluate nine defense configurations using a standardized pipeline with Qwen3-14B as teacher and Qwen2.5-7B-Instruct as student across three benchmarks (MATH-500, HumanEval+, MT-Bench). Our results reveal that, in a same-family distillation setting against a naive attacker, most output-level defenses are surprisingly ineffective: paraphrasing-based perturbation barely degrades distilled student quality, and data poisoning primarily impairs conversational fluency while leaving task-specific capabilities intact. Only chain-of-thought removal substantially impairs mathematical reasoning (31.4\% vs.\ 67.8\% baseline), though code generation remains unaffected. These findings demonstrate that the effectiveness of distillation defenses is highly task-dependent and that current output-level approaches are insufficient to broadly prevent knowledge theft.