Behavioral Fingerprints for LLM Endpoint Stability and Identity

TL;DR

This paper presents Stability Monitor, a black-box system that fingerprints AI model endpoints to detect behavioral changes over time, ensuring more reliable AI-native applications.

Contribution

It introduces a novel stability monitoring method using output distribution fingerprints and statistical tests to detect behavioral shifts in model endpoints.

Findings

Detects changes in model family, version, and inference stack

Identifies provider-to-provider stability differences

Successfully monitors behavioral stability in real-world settings

Abstract

The consistency of AI-native applications depends on the behavioral consistency of the model endpoints that power them. Traditional reliability metrics such as uptime, latency and throughput do not capture behavioral change, and an endpoint can remain "healthy" while its effective model identity changes due to updates to weights, tokenizers, quantization, inference engines, kernels, caching, routing, or hardware. We introduce Stability Monitor, a black-box stability monitoring system that periodically fingerprints an endpoint by sampling outputs from a fixed prompt set and comparing the resulting output distributions over time. Fingerprints are compared using a summed energy distance statistic across prompts, with permutation-test p-values as evidence of distribution shift aggregated sequentially to detect change events and define stability periods. In controlled validation, Stability Monitor detects changes to model family, version, inference stack, quantization, and behavioral parameters. In real-world monitoring of the same model hosted by multiple providers, we observe substantial provider-to-provider and within-provider stability differences.