Evaluating Reliability Gaps in Large Language Model Safety via Repeated Prompt Sampling

TL;DR

This paper introduces APST, a new depth-oriented evaluation method for LLM safety that tests models under repeated prompts to reveal latent failure modes and operational risks.

Contribution

The paper proposes APST, a stress testing framework inspired by reliability engineering, to assess LLM safety under repeated use and quantify failure probabilities.

Findings

Repeated sampling uncovers variability in failure rates across models and temperatures.

Shallow benchmarks may hide significant reliability differences in sustained use.

APST effectively surfaces latent safety failure modes in instruction-tuned LLMs.

Abstract

Traditional benchmarks for large language models (LLMs), such as HELM and AIR-BENCH, primarily assess safety risk through breadth-oriented evaluation across diverse tasks. However, real-world deployment often exposes a different class of risk: operational failures arising from repeated generations of the same prompt rather than broad task generalization. In high-stakes settings, response consistency and safety under repeated use are critical operational requirements. We introduce Accelerated Prompt Stress Testing (APST), a depth-oriented evaluation framework inspired by highly accelerated stress testing in reliability engineering. APST probes LLM behavior by repeatedly sampling identical prompts under controlled operational conditions, including temperature variation and prompt perturbation, to surface latent failure modes such as hallucinations, refusal inconsistency, and unsafe completions. Rather than treating failures as isolated events, APST characterizes them statistically as stochastic outcomes of repeated inference. We model observed safety failures using Bernoulli and binomial formulations to estimate per-inference failure probabilities, enabling quantitative comparison of operational risk across models and configurations. We apply APST to multiple instruction-tuned LLMs evaluated on AIR-BENCH 2024 derived safety and security prompts. While models exhibit similar performance under conventional single- or very-low-sample evaluation (N <= 3), repeated sampling reveals substantial variation in empirical failure probabilities across temperatures. These results demonstrate that shallow benchmark scores can obscure meaningful differences in reliability under sustained use.