Antidistillation Fingerprinting

TL;DR

The paper introduces antidistillation fingerprinting (ADFP), a new method that improves detection of model distillation by aligning fingerprinting with the student's learning process, outperforming existing techniques.

Contribution

ADFP is a principled, gradient-based approach that directly maximizes fingerprint detectability, achieving better detection with minimal utility loss across various tasks.

Findings

ADFP outperforms state-of-the-art baselines in detection confidence.

ADFP maintains high utility in mathematical reasoning, dialogue, and code generation.

Effective even when the student model's architecture is unknown.

Abstract

Model distillation enables efficient emulation of frontier large language models (LLMs), creating a need for robust mechanisms to detect when a third-party student model has trained on a teacher model's outputs. However, existing fingerprinting techniques that could be used to detect such distillation rely on heuristic perturbations that impose a steep trade-off between generation quality and fingerprinting strength, often requiring significant degradation of utility to ensure the fingerprint is effectively internalized by the student. We introduce antidistillation fingerprinting (ADFP), a principled approach that aligns the fingerprinting objective with the student's learning dynamics. Building upon the gradient-based framework of antidistillation sampling, ADFP utilizes a proxy model to identify and sample tokens that directly maximize the expected detectability of the fingerprint in the student after fine-tuning, rather than relying on the incidental absorption of the un-targeted biases of a more naive watermark. Experiments on GSM8K, OASST1, and MBPP demonstrate that ADFP achieves a significant Pareto improvement over state-of-the-art baselines, yielding stronger detection confidence with minimal impact on utility across mathematical reasoning, dialogue, and code generation, even when the student model's architecture is unknown.