When Punctuation Matters: A Large-Scale Comparison of Prompt Robustness Methods for LLMs

TL;DR

This paper systematically evaluates five prompt robustness methods across multiple LLMs and tasks, revealing their effectiveness and limitations against various distribution shifts to guide practitioners in enhancing model stability.

Contribution

It provides the first comprehensive comparison of prompt robustness techniques across diverse models and tasks within a unified framework, including frontier models like GPT-4.1.

Findings

Robustness methods vary significantly in effectiveness.

Fine-tuning and in-context learning approaches show different strengths.

Format perturbations impact model performance differently across models.

Abstract

Large Language Models (LLMs) are highly sensitive to subtle, non-semantic variations in prompt phrasing and formatting. In this work, we present the first systematic evaluation of 5 methods for improving prompt robustness within a unified experimental framework. We benchmark these techniques on 8 models from Llama, Qwen and Gemma families across 52 tasks from Natural Instructions dataset. Our evaluation covers robustness methods from both fine-tuned and in-context learning paradigms, and tests their generalization against multiple types of distribution shifts. Finally, we extend our analysis to GPT-4.1 and DeepSeek V3 to assess frontier models' current robustness to format perturbations. Our findings offer actionable insights into the relative effectiveness of these robustness methods, enabling practitioners to make informed decisions when aiming for stable and reliable LLM performance in real-world applications. Code: https://github.com/AIRI-Institute/when-punctuation-matters.