A Multi-Agent Framework for Feature-Constrained Difficulty Control in Reading Comprehension Item Generation

TL;DR

This paper presents MAFIG, a multi-agent framework utilizing collaborative LLMs and evaluators to generate reading comprehension items that reliably meet specified difficulty constraints.

Contribution

It introduces a multi-agent collaborative approach for feature-constrained item generation, improving difficulty adherence over single-agent methods.

Findings

MAFIG significantly outperforms baselines in adhering to feature constraints.

The method produces items with monotonically increasing difficulty levels.

Experimental results confirm robust difficulty control using the proposed framework.

Abstract

Recent studies in difficulty-controlled reading comprehension item generation have leveraged large language models (LLMs) to produce items by adjusting difficulty-related features. However, existing methods typically rely on a single-agent prompting approach, which often fails to consistently satisfy specified feature constraints, resulting in items that deviate from the target difficulty level. To address this limitation, we introduce MAFIG, a Multi-agent Framework for Feature-constrained Item Generation, where multiple LLM agents and feature-specific evaluators collaborate to generate and iteratively revise items based on intended constraints. Furthermore, to verify the efficacy of MAFIG in difficulty control, we propose a method for constructing a sequence of feature constraint sets that yield items with monotonically increasing difficulty. Experimental results demonstrate that MAFIG generates items that adhere to target constraints at a significantly higher rate than baselines, achieving robust difficulty control through the difficulty-calibrated constraint sequence.