Instruction Complexity Induces Positional Collapse in Adversarial LLM Evaluation

TL;DR

This study investigates how instruction complexity influences whether small instruction-tuned LLMs engage with content or rely on positional shortcuts during adversarial evaluation, revealing a spectrum of response behaviors.

Contribution

It uncovers the boundary conditions under which instruction complexity causes models to switch from content-aware to position-based shortcuts, highlighting the impact of multi-step instructions.

Findings

Vague instructions moderately reduce accuracy while maintaining content engagement.

Standard instructions induce positional entropy collapse with partial content sensitivity.

Multi-step instructions cause extreme positional collapse, with responses concentrated on a single position.

Abstract

When instructed to underperform on multiple-choice evaluations, do language models engage with question content or fall back on positional shortcuts? We map the boundary between these regimes using a six-condition adversarial instruction-specificity gradient administered to two instruction-tuned LLMs (Llama-3-8B and Llama-3.1-8B) on 2,000 MMLU-Pro items. Distributional screening (response-position entropy) and an independent content-engagement criterion (difficulty-accuracy correlation) jointly characterise each condition. The gradient reveals three regimes rather than a monotonic transition. Vague adversarial instructions produce moderate accuracy reduction with preserved content engagement. Standard sandbagging and capability-imitation instructions produce positional entropy collapse with partial content engagement. A two-step answer-aware avoidance instruction produces extreme positional collapse, with near-total concentration on a single response position (99.9% and 87.4%) and no measurable content sensitivity. This was the only multi-step instruction tested, and it produced the most extreme shortcut. The attractor position matches each model's content-absent null-prompt default. The effect replicates across both models and four academic domains. Distributional collapse and content engagement can co-occur (50% concordance between screening criteria), indicating that entropy-based screening and difficulty-based content assessment capture partially independent dimensions of response validity. Results suggest that instruction complexity can determine whether adversarial compliance uses content-aware or content-blind mechanisms in small instruction-tuned LLMs under greedy decoding.