Automated assessment of non-native learner essays: Investigating the role of linguistic features

TL;DR

This study investigates which linguistic features are most effective for automatic scoring of non-native English essays, revealing dataset-specific predictive features and demonstrating the importance of lexical, syntactic, discourse, and error features.

Contribution

It systematically evaluates the role of various linguistic features in AES across different datasets, highlighting dataset-specific predictive features and advancing understanding of feature importance.

Findings

Good predictive models achieved with the feature set

Most predictive features vary across datasets

Linguistic features include lexical, syntactic, discourse, and error types

Abstract

Automatic essay scoring (AES) refers to the process of scoring free text responses to given prompts, considering human grader scores as the gold standard. Writing such essays is an essential component of many language and aptitude exams. Hence, AES became an active and established area of research, and there are many proprietary systems used in real life applications today. However, not much is known about which specific linguistic features are useful for prediction and how much of this is consistent across datasets. This article addresses that by exploring the role of various linguistic features in automatic essay scoring using two publicly available datasets of non-native English essays written in test taking scenarios. The linguistic properties are modeled by encoding lexical, syntactic, discourse and error types of learner language in the feature set. Predictive models are then developed using these features on both datasets and the most predictive features are compared. While the results show that the feature set used results in good predictive models with both datasets, the question "what are the most predictive features?" has a different answer for each dataset.