Empirical Comparison of Encoder-Based Language Models and Feature-Based Supervised Machine Learning Approaches to Automated Scoring of Long Essays

TL;DR

This study compares encoder-based language models and feature-based supervised machine learning methods for automated long essay scoring, finding ensemble models of embeddings with gradient boosting outperform individual models.

Contribution

It introduces an ensemble-of-embeddings approach combining multiple language models with gradient boosting, improving long essay scoring accuracy.

Findings

Ensemble-of-embeddings model outperforms individual language models.

Gradient boosting enhances scoring performance.

Long essay scoring benefits from combining multiple model representations.

Abstract

Long context may impose challenges for encoder-only language models in text processing, specifically for automated scoring of essays. This study trained several commonly used encoder-based language models for automated scoring of long essays. The performance of these trained models was evaluated and compared with the ensemble models built upon the base language models with a token limit of 512?. The experimented models include BERT-based models (BERT, RoBERTa, DistilBERT, and DeBERTa), ensemble models integrating embeddings from multiple encoder models, and ensemble models of feature-based supervised machine learning models, including Gradient-Boosted Decision Trees, eXtreme Gradient Boosting, and Light Gradient Boosting Machine. We trained, validated, and tested each model on a dataset of 17,307 essays, with an 80%/10%/10% split, and evaluated model performance using Quadratic Weighted Kappa. This study revealed that an ensemble-of-embeddings model that combines multiple pre-trained language model representations with gradient-boosting classifier as the ensemble model significantly outperforms individual language models at scoring long essays.