Invisible Traces: Using Hybrid Fingerprinting to identify underlying LLMs in GenAI Apps

TL;DR

This paper presents a hybrid fingerprinting framework that combines static and dynamic analysis to accurately identify underlying LLMs in GenAI applications, even in complex, real-world scenarios with frequent updates and restricted access.

Contribution

It introduces a novel fingerprinting approach that addresses limitations of existing methods by integrating architectural and behavioral traits for robust LLM identification.

Findings

Effective in identifying LLMs in dynamic environments

Robust against frequent model updates and restricted access

Demonstrates high accuracy in simulated real-world conditions

Abstract

Fingerprinting refers to the process of identifying underlying Machine Learning (ML) models of AI Systemts, such as Large Language Models (LLMs), by analyzing their unique characteristics or patterns, much like a human fingerprint. The fingerprinting of Large Language Models (LLMs) has become essential for ensuring the security and transparency of AI-integrated applications. While existing methods primarily rely on access to direct interactions with the application to infer model identity, they often fail in real-world scenarios involving multi-agent systems, frequent model updates, and restricted access to model internals. In this paper, we introduce a novel fingerprinting framework designed to address these challenges by integrating static and dynamic fingerprinting techniques. Our approach identifies architectural features and behavioral traits, enabling accurate and robust fingerprinting of LLMs in dynamic environments. We also highlight new threat scenarios where traditional fingerprinting methods are ineffective, bridging the gap between theoretical techniques and practical application. To validate our framework, we present an extensive evaluation setup that simulates real-world conditions and demonstrate the effectiveness of our methods in identifying and monitoring LLMs in Gen-AI applications. Our results highlight the framework's adaptability to diverse and evolving deployment contexts.