Constructing Open Cloze Tests Using Generation and Discrimination Capabilities of Transformers

TL;DR

This paper introduces a multi-objective transformer model for creating open cloze tests, leveraging generation and discrimination capabilities, and demonstrates its effectiveness through experiments and expert evaluations.

Contribution

It presents the first multi-objective transformer approach for open cloze test construction, incorporating loss function tuning and re-ranking for improved test quality.

Findings

Achieves up to 82% accuracy in expert evaluations.

Outperforms previous methods and baselines.

Provides a new benchmark dataset for future research.

Abstract

This paper presents the first multi-objective transformer model for constructing open cloze tests that exploits generation and discrimination capabilities to improve performance. Our model is further enhanced by tweaking its loss function and applying a post-processing re-ranking algorithm that improves overall test structure. Experiments using automatic and human evaluation show that our approach can achieve up to 82% accuracy according to experts, outperforming previous work and baselines. We also release a collection of high-quality open cloze tests along with sample system output and human annotations that can serve as a future benchmark.