Do Stop Me Now: Detecting Boilerplate Responses with a Single Iteration

TL;DR

This paper introduces a simple method to detect boilerplate responses in large language models after just one generation step, enabling early termination and reducing computational costs.

Contribution

The paper proposes using the first token's log-probability distribution as a signal to classify responses, demonstrating high accuracy across various models with minimal computation.

Findings

High accuracy in detecting boilerplate responses using first-token log-probabilities

Distinct clustering of response types across different models

Significant computational savings through early response classification

Abstract

Large Language Models (LLMs) often expend significant computational resources generating boilerplate responses, such as refusals, simple acknowledgements and casual greetings, which adds unnecessary cost and latency. To address this inefficiency, we propose a simple yet highly effective method for detecting such responses after only a single generation step. We demonstrate that the log-probability distribution of the first generated token serves as a powerful signal for classifying the nature of the entire subsequent response. Our experiments, conducted across a diverse range of small, large, and reasoning-specialized models, show that the first-token log-probability vectors form distinctly separable clusters for different response types. Using a lightweight k-NN classifier, we achieve high accuracy in predicting whether a response will be a substantive answer or a form of boilerplate response, including user-specified refusals. The primary implication is a practical, computationally trivial technique, optimizing LLM inference by enabling early termination or redirection to a smaller model, thereby yielding significant savings in computational cost. This work presents a direct path toward more efficient and sustainable LLM deployment.