ErrEval: Error-Aware Evaluation for Question Generation through Explicit Diagnostics

TL;DR

ErrEval introduces an error-aware evaluation framework for question generation that explicitly diagnoses common errors to improve evaluation accuracy and alignment with human judgments.

Contribution

The paper presents ErrEval, a novel framework that incorporates explicit error diagnostics into QG evaluation, addressing limitations of existing black-box methods.

Findings

ErrEval improves correlation with human judgments.

Explicit diagnostics mitigate overestimation of low-quality questions.

Demonstrated effectiveness on three benchmark datasets.

Abstract

Automatic Question Generation (QG) often produces outputs with critical defects, such as factual hallucinations and answer mismatches. However, existing evaluation methods, including LLM-based evaluators, mainly adopt a black-box and holistic paradigm without explicit error modeling, leading to the neglect of such defects and overestimation of question quality. To address this issue, we propose ErrEval, a flexible and Error-aware Evaluation framework that enhances QG evaluation through explicit error diagnostics. Specifically, ErrEval reformulates evaluation as a two-stage process of error diagnosis followed by informed scoring. At the first stage, a lightweight plug-and-play Error Identifier detects and categorizes common errors across structural, linguistic, and content-related aspects. These diagnostic signals are then incorporated as explicit evidence to guide LLM evaluators toward more fine-grained and grounded judgments. Extensive experiments on three benchmarks demonstrate the effectiveness of ErrEval, showing that incorporating explicit diagnostics improves alignment with human judgments. Further analyses confirm that ErrEval effectively mitigates the overestimation of low-quality questions.