On the Paradoxical Interference between Instruction-Following and Task Solving

TL;DR

This paper uncovers a paradoxical interference where instruction following can hinder LLMs' task-solving abilities, introduces a metric to measure this effect, and analyzes its causes and implications across models and constraints.

Contribution

It reveals the counterintuitive interference between instruction following and task solving in LLMs, and proposes SUSTAINSCORE to quantify and analyze this phenomenon.

Findings

Adding self-evident constraints causes performance drops in LLMs.

Interference is consistent across different constraint types and model scales.

Failed cases show more attention to constraints than successful ones.

Abstract

Instruction following aims to align Large Language Models (LLMs) with human intent by specifying explicit constraints on how tasks should be performed. However, we reveal a counterintuitive phenomenon: instruction following can paradoxically interfere with LLMs' task-solving capability. We propose a metric, SUSTAINSCORE, to quantify the interference of instruction following with task solving. It measures task performance drop after inserting into the instruction a self-evident constraint, which is naturally met by the original successful model output and extracted from it. Experiments on current LLMs in mathematics, multi-hop QA, and code generation show that adding the self-evident constraints leads to substantial performance drops, even for advanced models such as Claude-Sonnet-4.5. We validate the generality of the interference across constraint types and scales. Furthermore, we identify common failure patterns, and by investigating the mechanisms of interference, we observe that failed cases allocate significantly more attention to constraints compared to successful ones. Finally, we use SUSTAINSCORE to conduct an initial investigation into how distinct post-training paradigms affect the interference, presenting empirical observations on current alignment strategies. We will release our code and data to facilitate further research