RIFT: Reordered Instruction Following Testbed To Evaluate Instruction Following in Singular Multistep Prompt Structures
Andrew Jaffe, Noah Reicin, Jinho D. Choi

TL;DR
This paper introduces RIFT, a benchmark to evaluate how well large language models follow instructions in different prompt structures, revealing significant sensitivity to prompt order and structure.
Contribution
The paper presents RIFT, a novel benchmark that isolates prompt structure effects on instruction following, highlighting the structural sensitivity of current LLMs.
Findings
Accuracy drops up to 72% in jumping prompts
Approximately 50% of failures due to instruction-order violations
Current models treat instruction following as sequential pattern learning
Abstract
Large Language Models (LLMs) are increasingly relied upon for complex workflows, yet their ability to maintain flow of instructions remains underexplored. Existing benchmarks conflate task complexity with structural ordering, making it difficult to isolate the impact of prompt topology on performance. We introduce RIFT, Reordered Instruction Following Testbed, to assess instruction following by disentangling structure from content. Using rephrased Jeopardy! question-answer pairs, we test LLMs across two prompt structures: linear prompts, which progress sequentially, and jumping prompts, which preserve identical content but require non-sequential traversal. Across 10,000 evaluations spanning six state-of-the-art open-source LLMs, accuracy dropped by up to 72% under jumping conditions (compared to baseline), revealing a strong dependence on positional continuity. Error analysis shows that…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsTopic Modeling · Natural Language Processing Techniques · Software System Performance and Reliability
