Language Model Maps for Prompt-Response Distributions via Log-Likelihood Vectors

TL;DR

This paper introduces a novel method for representing and comparing language models using log-likelihood vectors over prompt-response pairs, enabling analysis of model relationships, prompt effects, and input-dependent behaviors.

Contribution

The paper presents a new framework for mapping language models in a space where distances approximate KL divergence, capturing systematic prompt effects and improving interpretability.

Findings

Model maps reflect meaningful global structure and model attributes.

Prompt modifications induce systematic shifts that are approximately additive.

PMI vectors can better highlight training-data-related differences.

Abstract

We propose a method that represents language models by log-likelihood vectors over prompt-response pairs and constructs model maps for comparing their conditional distributions. In this space, distances between models approximate the KL divergence between the corresponding conditional distributions. Experiments on a large collection of publicly available language models show that the maps capture meaningful global structure, including relationships to model attributes and task performance. The method also captures systematic shifts induced by prompt modifications and their approximate additive compositionality, suggesting a way to analyze and predict the effects of composite prompt operations. We further introduce pointwise mutual information (PMI) vectors to reduce the influence of unconditional distributions; in some cases, PMI-based model maps better reflect training-data-related differences. Overall, the framework supports the analysis of input-dependent model behavior.