On the Evaluation of Answer-Agnostic Paragraph-level Multi-Question Generation

TL;DR

This paper introduces a novel evaluation method for answer-agnostic paragraph-level multi-question generation and compares various strategies for question generation using pre-trained seq2seq models.

Contribution

It proposes a new evaluation approach using the Hungarian algorithm and analyzes different question generation strategies with pre-trained models.

Findings

The Hungarian algorithm improves question set evaluation accuracy.

Pre-trained seq2seq models can effectively generate relevant questions.

Evaluation method better accounts for reference coverage.

Abstract

We study the task of predicting a set of salient questions from a given paragraph without any prior knowledge of the precise answer. We make two main contributions. First, we propose a new method to evaluate a set of predicted questions against the set of references by using the Hungarian algorithm to assign predicted questions to references before scoring the assigned pairs. We show that our proposed evaluation strategy has better theoretical and practical properties compared to prior methods because it can properly account for the coverage of references. Second, we compare different strategies to utilize a pre-trained seq2seq model to generate and select a set of questions related to a given paragraph. The code is available.