MulDimIF: A Multi-Dimensional Constraint Framework for Evaluating and Improving Instruction Following in Large Language Models

TL;DR

This paper introduces MulDimIF, a multi-dimensional constraint framework for evaluating and enhancing instruction-following in large language models, demonstrating significant performance improvements through targeted training.

Contribution

The paper presents a novel multi-dimensional constraint framework and a controllable instruction generation pipeline, enabling systematic evaluation and improvement of LLMs' instruction-following abilities.

Findings

Performance drops from 80.82% to 36.76% across constraint levels.

Training with generated data improves instruction following significantly.

Attention modules are key to better constraint recognition and adherence.

Abstract

Instruction following refers to the ability of large language models (LLMs) to generate outputs that satisfy all specified constraints. Existing research has primarily focused on constraint categories, offering limited evaluation dimensions and little guidance for improving instruction-following abilities. To address this gap, we introduce MulDimIF, a multi-dimensional constraint framework encompassing three constraint patterns, four constraint categories, and four difficulty levels. Based on this framework, we design a controllable instruction generation pipeline. Through constraint expansion, conflict detection, and instruction rewriting, we construct 9,106 code-verifiable samples. We evaluate 18 LLMs from six model families and find marked performance differences across constraint settings. For instance, average accuracy decreases from 80.82% at Level I to 36.76% at Level IV. Moreover, training with data generated by our framework significantly improves instruction following without compromising general performance. In-depth analysis indicates that these gains stem largely from parameter updates in attention modules, which strengthen constraint recognition and adherence. Code and data are available in https://github.com/Junjie-Ye/MulDimIF.