ConInstruct: Evaluating Large Language Models on Conflict Detection and Resolution in Instructions

TL;DR

This paper introduces ConInstruct, a benchmark for evaluating large language models' ability to detect and resolve conflicting instructions, revealing strengths in conflict detection but weaknesses in user notification and clarification.

Contribution

The paper presents a new benchmark, ConInstruct, to assess LLMs' conflict detection and resolution, filling a gap in evaluating their behavior with conflicting instructions.

Findings

Most proprietary LLMs detect conflicts well.

Open-source models vary in conflict detection performance.

LLMs rarely notify users or seek clarification about conflicts.

Abstract

Instruction-following is a critical capability of Large Language Models (LLMs). While existing works primarily focus on assessing how well LLMs adhere to user instructions, they often overlook scenarios where instructions contain conflicting constraints-a common occurrence in complex prompts. The behavior of LLMs under such conditions remains under-explored. To bridge this gap, we introduce ConInstruct, a benchmark specifically designed to assess LLMs' ability to detect and resolve conflicts within user instructions. Using this dataset, we evaluate LLMs' conflict detection performance and analyze their conflict resolution behavior. Our experiments reveal two key findings: (1) Most proprietary LLMs exhibit strong conflict detection capabilities, whereas among open-source models, only DeepSeek-R1 demonstrates similarly strong performance. DeepSeek-R1 and Claude-4.5-Sonnet achieve the highest average F1-scores at 91.5% and 87.3%, respectively, ranking first and second overall. (2) Despite their strong conflict detection abilities, LLMs rarely explicitly notify users about the conflicts or request clarification when faced with conflicting constraints. These results underscore a critical shortcoming in current LLMs and highlight an important area for future improvement when designing instruction-following LLMs.