DuFFin: A Dual-Level Fingerprinting Framework for LLMs IP Protection

TL;DR

DuFFin is a dual-level fingerprinting framework designed to protect the intellectual property of large language models by accurately verifying ownership in black-box settings through trigger patterns and knowledge-level fingerprints.

Contribution

It introduces a novel dual-level fingerprinting approach that effectively verifies LLM ownership without impacting text generation or requiring white-box access.

Findings

Achieves IP-ROC > 0.95 in experiments

Works across various model types and modifications

Effective in black-box ownership verification

Abstract

Large language models (LLMs) are considered valuable Intellectual Properties (IP) for legitimate owners due to the enormous computational cost of training. It is crucial to protect the IP of LLMs from malicious stealing or unauthorized deployment. Despite existing efforts in watermarking and fingerprinting LLMs, these methods either impact the text generation process or are limited in white-box access to the suspect model, making them impractical. Hence, we propose DuFFin, a novel $\textbf{Du}$al-Level $\textbf{Fin}$gerprinting $\textbf{F}$ramework for black-box setting ownership verification. DuFFin extracts the trigger pattern and the knowledge-level fingerprints to identify the source of a suspect model. We conduct experiments on a variety of models collected from the open-source website, including four popular base models as protected LLMs and their fine-tuning, quantization, and safety alignment versions, which are released by large companies, start-ups, and individual users. Results show that our method can accurately verify the copyright of the base protected LLM on their model variants, achieving the IP-ROC metric greater than 0.95. Our code is available at https://github.com/yuliangyan0807/llm-fingerprint.