QRelScore: Better Evaluating Generated Questions with Deeper Understanding of Context-aware Relevance

TL;DR

QRelScore is a novel context-aware metric for evaluating question generation that better aligns with human judgment by considering input context and reasoning complexity.

Contribution

It introduces QRelScore, a new evaluation metric utilizing language models for deeper context understanding and reasoning in question relevance assessment.

Findings

QRelScore correlates more strongly with human judgments than existing metrics.

It demonstrates robustness against adversarial samples.

It effectively handles complex reasoning and multiple evidences.

Abstract

Existing metrics for assessing question generation not only require costly human reference but also fail to take into account the input context of generation, rendering the lack of deep understanding of the relevance between the generated questions and input contexts. As a result, they may wrongly penalize a legitimate and reasonable candidate question when it (i) involves complicated reasoning with the context or (ii) can be grounded by multiple evidences in the context. In this paper, we propose $\textbf{QRelScore}$, a context-aware $\underline{\textbf{Rel}}$evance evaluation metric for $\underline{\textbf{Q}}$uestion Generation. Based on off-the-shelf language models such as BERT and GPT2, QRelScore employs both word-level hierarchical matching and sentence-level prompt-based generation to cope with the complicated reasoning and diverse generation from multiple evidences, respectively. Compared with existing metrics, our experiments demonstrate that QRelScore is able to achieve a higher correlation with human judgments while being much more robust to adversarial samples.