Deconstructing Instruction-Following: A New Benchmark for Granular Evaluation of Large Language Model Instruction Compliance Abilities

TL;DR

This paper introduces MOSAIC, a modular benchmark for detailed evaluation of LLMs' ability to follow complex instructions, revealing variability in compliance across different constraints and model types.

Contribution

The paper presents MOSAIC, a novel framework for granular, independent assessment of instruction compliance in LLMs, addressing limitations of existing benchmarks.

Findings

Compliance varies with constraint type, quantity, and position.

Model-specific weaknesses and biases such as primacy and recency effects.

Interactions between instructions can be synergistic or conflicting.

Abstract

Reliably ensuring Large Language Models (LLMs) follow complex instructions is a critical challenge, as existing benchmarks often fail to reflect real-world use or isolate compliance from task success. We introduce MOSAIC (MOdular Synthetic Assessment of Instruction Compliance), a modular framework that uses a dynamically generated dataset with up to 20 application-oriented generation constraints to enable a granular and independent analysis of this capability. Our evaluation of five LLMs from different families based on this new benchmark demonstrates that compliance is not a monolithic capability but varies significantly with constraint type, quantity, and position. The analysis reveals model-specific weaknesses, uncovers synergistic and conflicting interactions between instructions, and identifies distinct positional biases such as primacy and recency effects. These granular insights are critical for diagnosing model failures and developing more reliable LLMs for systems that demand strict adherence to complex instructions.