Scalable Fingerprinting of Large Language Models

TL;DR

This paper introduces a scalable fingerprinting method for large language models, enabling the embedding of thousands of unique identifiers without affecting model utility, and demonstrating robustness against fine-tuning and security threats.

Contribution

We propose Perinucleus sampling, a novel scalable fingerprinting technique that embeds thousands of persistent, harmless fingerprints into large language models.

Findings

Can embed 24,576 fingerprints into a Llama-3.1-8B model

Fingerprints remain effective after fine-tuning

Scheme mitigates security risks associated with fingerprinting

Abstract

Model fingerprinting has emerged as a powerful tool for model owners to identify their shared model given API access. However, to lower false discovery rate, fight fingerprint leakage, and defend against coalitions of model users attempting to bypass detection, we argue that {\em scalability} is critical, i.e., scaling up the number of fingerprints one can embed into a model. Hence, we pose scalability as a crucial requirement for fingerprinting schemes. We experiment with fingerprint design at a scale significantly larger than previously considered, and introduce a new method, dubbed Perinucleus sampling, to generate scalable, persistent, and harmless fingerprints. We demonstrate that this scheme can add 24,576 fingerprints to a Llama-3.1-8B model -- two orders of magnitude more than existing schemes -- without degrading the model's utility. Our inserted fingerprints persist even after supervised fine-tuning on standard post-training data. We further address security risks for fingerprinting, and theoretically and empirically show how a scalable fingerprinting scheme like ours can mitigate these risks. Our code is available at https://github.com/SewoongLab/scalable-fingerprinting-of-llms