Automated Text Scoring in the Age of Generative AI for the GPU-poor

TL;DR

This paper evaluates the potential of small, open-source generative language models for automated text scoring on modest hardware, highlighting their performance, efficiency, and capacity to generate feedback, as alternatives to proprietary models.

Contribution

It explores the use of open-source, small-scale GLMs for ATS, demonstrating their fine-tuning capabilities and initial steps towards automated feedback generation.

Findings

GLMs can be fine-tuned for adequate ATS performance

Open-source models work on modest hardware

Model-generated feedback shows potential but needs further validation

Abstract

Current research on generative language models (GLMs) for automated text scoring (ATS) has focused almost exclusively on querying proprietary models via Application Programming Interfaces (APIs). Yet such practices raise issues around transparency and security, and these methods offer little in the way of efficiency or customizability. With the recent proliferation of smaller, open-source models, there is the option to explore GLMs with computers equipped with modest, consumer-grade hardware, that is, for the "GPU poor." In this study, we analyze the performance and efficiency of open-source, small-scale GLMs for ATS. Results show that GLMs can be fine-tuned to achieve adequate, though not state-of-the-art, performance. In addition to ATS, we take small steps towards analyzing models' capacity for generating feedback by prompting GLMs to explain their scores. Model-generated feedback shows promise, but requires more rigorous evaluation focused on targeted use cases.