CASTILLO: Characterizing Response Length Distributions of Large Language Models

TL;DR

CASTILLO provides a comprehensive dataset and analysis of response length distributions across multiple large language models, revealing variability and aiding resource management in LLM inference.

Contribution

It introduces a dataset characterizing response lengths for 13 LLMs across diverse prompts, enabling better predictive modeling and resource allocation.

Findings

Significant variability in response lengths within and across models

Model-specific behaviors and partial text degeneration observed

Dataset and analysis tools publicly released for research use

Abstract

Efficiently managing compute resources for Large Language Model (LLM) inference remains challenging due to the inherently stochastic and variable lengths of autoregressive text generation. Accurately estimating response lengths in advance enables proactive resource allocation, yet existing approaches either bias text generation towards certain lengths or rely on assumptions that ignore model- and prompt-specific variability. We introduce CASTILLO, a dataset characterizing response length distributions across 13 widely-used open-source LLMs evaluated on seven distinct instruction-following corpora. For each $\langle$prompt, model$\rangle$ sample pair, we generate 10 independent completions using fixed decoding hyper-parameters, record the token length of each response, and publish summary statistics (mean, std-dev, percentiles), along with the shortest and longest completions, and the exact generation settings. Our analysis reveals significant inter- and intra-model variability in response lengths (even under identical generation settings), as well as model-specific behaviors and occurrences of partial text degeneration in only subsets of responses. CASTILLO enables the development of predictive models for proactive scheduling and provides a systematic framework for analyzing model-specific generation behaviors. We publicly release the dataset and code to foster research at the intersection of generative language modeling and systems.