AGReE: A system for generating Automated Grammar Reading Exercises

TL;DR

The paper introduces AGReE, a system that automatically generates grammar reading exercises from user passages, including multiple-choice questions for various grammar topics, validated by large-scale human evaluation.

Contribution

It presents a novel automated system for generating grammar exercises with high accuracy, validated through extensive human evaluation, covering multiple grammar constructs.

Findings

95% of items had a majority of raters identify the correct answer

85% of cases had raters agree on a single correct answer

Most mistakes occurred in punctuation and conjunction exercises

Abstract

We describe the AGReE system, which takes user-submitted passages as input and automatically generates grammar practice exercises that can be completed while reading. Multiple-choice practice items are generated for a variety of different grammar constructs: punctuation, articles, conjunctions, pronouns, prepositions, verbs, and nouns. We also conducted a large-scale human evaluation with around 4,500 multiple-choice practice items. We notice for 95% of items, a majority of raters out of five were able to identify the correct answer and for 85% of cases, raters agree that there is only one correct answer among the choices. Finally, the error analysis shows that raters made the most mistakes for punctuation and conjunctions.