Visual Fingerprints for LLM Generation Comparison

TL;DR

This paper introduces a visual fingerprinting method to compare large language model outputs across different generation conditions by analyzing linguistic choice distributions.

Contribution

It presents a novel visualization approach that models and compares LLM responses as distributions of linguistic features across varied settings.

Findings

Visual fingerprints reveal consistent patterns in LLM behavior.

The method enables comparison of responses across different generation conditions.

It helps identify biases and tendencies in LLM outputs.

Abstract

Large language model (LLM) outputs arise from complex interactions among prompts, system instructions, model parameters, and architecture. We refer to specific configurations of these factors as generation conditions, each of which can bias outputs in various ways. Understanding how different generation conditions shape model behaviors is essential for tasks such as prompt design and model evaluation, yet it remains challenging due to the stochastic and open-ended nature of text generation. We present an approach to visually compare LLM outputs across generation conditions by modeling responses as collections of linguistic choices, including content, expression, and structure. We extract these choices using natural language processing pipelines and represent their distributions across repeated samples. We then visualize these distributions as visual fingerprints, enabling direct, distribution-level comparison of condition-specific tendencies. Through four usage scenarios, we demonstrate how visual fingerprints reveal consistent patterns in LLM behavior that are difficult to observe through individual responses or aggregate metrics.