Hide and Seek: Fingerprinting Large Language Models with Evolutionary Learning

TL;DR

This paper presents a novel black-box method using evolutionary learning and LLMs to fingerprint and identify different large language models with 72% accuracy, enhancing AI transparency and security.

Contribution

It introduces a unique 'Hide and Seek' algorithm leveraging LLMs for model fingerprinting, combining evolutionary strategies with in-context learning for improved identification.

Findings

Achieved 72% accuracy in identifying LLM families

Developed a 'Hide and Seek' algorithm for model fingerprinting

Revealed insights into semantic differences among LLMs

Abstract

As content generated by Large Language Model (LLM) has grown exponentially, the ability to accurately identify and fingerprint such text has become increasingly crucial. In this work, we introduce a novel black-box approach for fingerprinting LLMs, achieving an impressive 72% accuracy in identifying the correct family of models (Such as Llama, Mistral, Gemma, etc) among a lineup of LLMs. We present an evolutionary strategy that leverages the capabilities of one LLM to discover the most salient features for identifying other LLMs. Our method employs a unique "Hide and Seek" algorithm, where an Auditor LLM generates discriminative prompts, and a Detective LLM analyzes the responses to fingerprint the target models. This approach not only demonstrates the feasibility of LLM-driven model identification but also reveals insights into the semantic manifolds of different LLM families. By iteratively refining prompts through in-context learning, our system uncovers subtle distinctions between model outputs, providing a powerful tool for LLM analysis and verification. This research opens new avenues for understanding LLM behavior and has significant implications for model attribution, security, and the broader field of AI transparency.