Syntactic Framing Fragility: An Audit of Robustness in LLM Ethical Decisions

TL;DR

This paper introduces Syntactic Framing Fragility (SFF), a framework to measure how large language models' ethical decisions vary with syntactic changes, revealing significant negation sensitivity especially in open-source models.

Contribution

The paper presents SFF, a novel operational framework and metric for quantifying LLM decision robustness against syntactic variations, enabling scalable auditing.

Findings

Open-source models are 2.2x more fragile than commercial models.

Negation syntax is the main failure mode affecting decision consistency.

Chain-of-thought reasoning can reduce fragility in some cases.

Abstract

Large language models exhibit systematic negation sensitivity, yet no operational framework exists to measure this vulnerability at deployment scale, especially in high-stakes decisions. We introduce Syntactic Framing Fragility (SFF), a framework for quantifying decision consistency under logically equivalent syntactic transformations. SFF isolates syntactic effects via Logical Polarity Normalization, enabling direct comparison across positive and negative framings while controlling for polarity inversion, and provides the Syntactic Variation Index (SVI) as a robustness metric suitable for CI/CD integration. Auditing 23 models across 14 high-stakes scenarios (39,975 decisions), we establish ground-truth effect sizes for a phenomenon previously characterized only qualitatively and find that open-source models exhibit $2.2x higher fragility than commercial counterparts. Negation-bearing syntax is the dominant failure mode, with some models endorsing actions at 80-97% rates even when asked whether agents not act. These patterns are consistent with negation suppression failure documented in prior work, with chain-of-thought reasoning reducing fragility in some but not all cases. We provide scenario-stratified risk profiles and offer an operational checklist compatible with EU AI Act and NIST RMF requirements. Code, data, and scenarios will be released upon publication.