LLMmap: Fingerprinting For Large Language Models

TL;DR

LLMmap is a novel fingerprinting method that accurately identifies specific large language model versions using minimal interactions, robust across various application settings and resistant to countermeasures.

Contribution

This paper introduces LLMmap, the first active fingerprinting technique capable of identifying numerous LLM versions with high accuracy across diverse application environments.

Findings

Achieves over 95% accuracy with 8 queries

Identifies 42 LLM versions including open-source and proprietary

Robust against different system prompts and sampling hyperparameters

Abstract

We introduce LLMmap, a first-generation fingerprinting technique targeted at LLM-integrated applications. LLMmap employs an active fingerprinting approach, sending carefully crafted queries to the application and analyzing the responses to identify the specific LLM version in use. Our query selection is informed by domain expertise on how LLMs generate uniquely identifiable responses to thematically varied prompts. With as few as 8 interactions, LLMmap can accurately identify 42 different LLM versions with over 95% accuracy. More importantly, LLMmap is designed to be robust across different application layers, allowing it to identify LLM versions--whether open-source or proprietary--from various vendors, operating under various unknown system prompts, stochastic sampling hyperparameters, and even complex generation frameworks such as RAG or Chain-of-Thought. We discuss potential mitigations and demonstrate that, against resourceful adversaries, effective countermeasures may be challenging or even unrealizable.