Measuring Pragmatic Influence in Large Language Model Instructions

TL;DR

This paper introduces a framework to systematically measure how pragmatic framing cues in instructions influence large language model behavior, revealing consistent shifts in directive prioritization across various models.

Contribution

It presents a novel decomposition, taxonomy, and measurement method for pragmatic framing effects, establishing them as measurable factors in instruction following.

Findings

Pragmatic framing causes predictable shifts in model directive prioritization.

Influence mechanisms are consistent across different LLMs.

The framework enables systematic analysis of framing effects.

Abstract

It is not only what we ask large language models (LLMs) to do that matters, but also how we prompt. Phrases like "This is urgent" or "As your supervisor" can shift model behavior without altering task content. We study this effect as pragmatic framing, contextual cues that shape directive interpretation rather than task specification. While prior work exploits such cues for prompt optimization or probes them as security vulnerabilities, pragmatic framing itself has not been treated as a measurable property of instruction following. Measuring this influence systematically remains challenging, requiring controlled isolation of framing cues. We introduce a framework with three novel components: directive-framing decomposition separating framing context from task specification; a taxonomy organizing 400 instantiations of framing into 13 strategies across 4 mechanism clusters; and priority-based measurement that quantifies influence through observable shifts in directive prioritization. Across five LLMs of different families and sizes, influence mechanisms cause consistent and structured shifts in directive prioritization, moving models from baseline impartiality toward favoring the framed directive. This work establishes pragmatic framing as a measurable and predictable factor in instruction-following systems.