Measuring Distribution Shift in User Prompts and Its Effects on LLM Performance

TL;DR

This paper introduces the LENS framework to measure how natural prompt distribution shifts affect the performance of deployed large language models, revealing significant performance drops across diverse user groups and over time.

Contribution

The paper presents a large-scale, data-centric evaluation method for quantifying prompt distribution shifts and their impact on LLM performance in real-world settings.

Findings

Moderate prompt shifts cause 73% average performance loss in LLMs.

Performance degradation is more severe across different user groups and regions.

Prompt distribution shifts over time are strongly correlated with LLM performance drops.

Abstract

LLMs are increasingly deployed in dynamic, real-world settings, where the distribution of user prompts can shift substantially over time as new tasks, prompts, and users are introduced to a deployed model. Such natural prompt distribution shift poses a major challenge to LLM reliability, particularly for specialized models designed for narrow domains or user populations. Despite attention to out-of-distribution robustness, there is very limited exploration of measuring natural prompt distribution shift in prior work, and its impact on deployed LLMs remains poorly understood. We introduce the LLM Evaluation under Natural prompt Shift (LENS) framework: a data-centric approach for quantifying natural prompt distribution shift and evaluating its effect on the performance of deployed LLMs. We perform a large-scale evaluation using 192 real-world post-deployment prompt shift settings over time, user group, and geographic axes, training a total of 81 models on 4.68M training prompts, and evaluating on 57.6k prompts. We find that even moderate shifts in user prompt behavior correspond with large performance drops (73% average loss) in deployed LLMs. This performance degradation is particularly prevalent when users from different latent groups and geographic regions interact with models and is correlated with natural prompt distribution shift over time. We systematically characterize how LLM instruction following ability degrades over time and between user groups. Our findings highlight the critical need for data-driven monitoring to ensure LLM performance remains stable across diverse and evolving user populations.