Towards Distillation-Resistant Large Language Models: An Information-Theoretic Perspective

TL;DR

This paper introduces an information-theoretic approach to defend large language models against knowledge distillation attacks by minimizing the distillation-relevant information in their outputs.

Contribution

It proposes a novel CMI-based method to transform model outputs, reducing their vulnerability to distillation without sacrificing task performance.

Findings

The method significantly reduces distillation success across multiple LLMs.

It preserves task accuracy while decreasing extractability of model knowledge.

The approach is effective against various distillation algorithms.

Abstract

Proprietary large language models (LLMs) embody substantial economic value and are generally exposed only as black-box APIs, yet adversaries can still exploit their outputs to extract knowledge via distillation. Existing defenses focus exclusively on text-based distillation, leaving the important logit-based distillation largely unexplored. In this work, we analyze this problem and present an effective solution from an information-theoretic perspective. We characterize distillation-relevant information in teacher outputs using the conditional mutual information (CMI) between teacher logits and input queries conditioned on ground-truth labels. This quantity captures contextual information beneficial for model extraction, motivating us to defend distillation via CMI minimization. Guided by our theoretical analysis, we propose learning a transformation matrix that purifies the original outputs to enhance distillation resistance. We further derive a CMI-inspired anti-distillation objective to optimize this transformation, which effectively removes distillation-relevant information while preserving output utility. Extensive experiments across multiple LLMs and strong distillation algorithms demonstrate that the proposed method significantly degrades distillation performance while preserving task accuracy, effectively protecting models' intellectual property.