Say It Another Way: Auditing LLMs with a User-Grounded Automated Paraphrasing Framework

TL;DR

AUGMENT is a framework that generates controlled, user-grounded paraphrases to improve the reliability of auditing large language models by uncovering systematic weaknesses.

Contribution

We introduce AUGMENT, a novel framework for producing linguistically informed, controlled paraphrases grounded in user behavior, enhancing LLM auditing accuracy.

Findings

Controlled paraphrases reveal weaknesses hidden under unconstrained variation.

AUGMENT improves the reliability of LLM auditing.

Case studies demonstrate effectiveness on BBQ and MMLU datasets.

Abstract

Large language models (LLMs) are highly sensitive to subtle changes in prompt phrasing, posing challenges for reliable auditing. Prior methods often apply unconstrained prompt paraphrasing, which risk missing linguistic and demographic factors that shape authentic user interactions. We introduce AUGMENT (Automated User-Grounded Modeling and Evaluation of Natural Language Transformations), a framework for generating controlled paraphrases, grounded in user behaviors. AUGMENT leverages linguistically informed rules and enforces quality through checks on instruction adherence, semantic similarity, and realism, ensuring paraphrases are both reliable and meaningful for auditing. Through case studies on the BBQ and MMLU datasets, we show that controlled paraphrases uncover systematic weaknesses that remain obscured under unconstrained variation. These results highlight the value of the AUGMENT framework for reliable auditing.