Enhancing Essay Cohesion Assessment: A Novel Item Response Theory Approach

TL;DR

This paper introduces a novel application of item response theory to improve automatic cohesion scoring in essays, demonstrating enhanced performance over traditional machine learning methods on Brazilian educational datasets.

Contribution

It adapts item response theory to the context of essay cohesion assessment, providing a new method that adjusts machine learning scores for better accuracy.

Findings

Proposed approach outperforms conventional models in multiple metrics

Effective in evaluating essays from diverse Brazilian educational datasets

Enhances automatic cohesion scoring accuracy in educational assessments

Abstract

Essays are considered a valuable mechanism for evaluating learning outcomes in writing. Textual cohesion is an essential characteristic of a text, as it facilitates the establishment of meaning between its parts. Automatically scoring cohesion in essays presents a challenge in the field of educational artificial intelligence. The machine learning algorithms used to evaluate texts generally do not consider the individual characteristics of the instances that comprise the analysed corpus. In this meaning, item response theory can be adapted to the context of machine learning, characterising the ability, difficulty and discrimination of the models used. This work proposes and analyses the performance of a cohesion score prediction approach based on item response theory to adjust the scores generated by machine learning models. In this study, the corpus selected for the experiments consisted of the extended Essay-BR, which includes 6,563 essays in the style of the National High School Exam (ENEM), and the Brazilian Portuguese Narrative Essays, comprising 1,235 essays written by 5th to 9th grade students from public schools. We extracted 325 linguistic features and treated the problem as a machine learning regression task. The experimental results indicate that the proposed approach outperforms conventional machine learning models and ensemble methods in several evaluation metrics. This research explores a potential approach for improving the automatic evaluation of cohesion in educational essays.