Balancing Cost and Quality: An Exploration of Human-in-the-loop Frameworks for Automated Short Answer Scoring

TL;DR

This paper explores a human-in-the-loop framework for automated short answer scoring that combines deep learning models with human graders, aiming to reduce grading costs while maintaining high quality through confidence estimation.

Contribution

It introduces a confidence estimation method enabling SAS models to selectively defer uncertain cases to human graders, balancing cost and quality.

Findings

The framework achieves target scoring quality with reduced human grading effort.

Confidence estimation effectively identifies predictions requiring human review.

Multiple confidence methods were validated across various datasets.

Abstract

Short answer scoring (SAS) is the task of grading short text written by a learner. In recent years, deep-learning-based approaches have substantially improved the performance of SAS models, but how to guarantee high-quality predictions still remains a critical issue when applying such models to the education field. Towards guaranteeing high-quality predictions, we present the first study of exploring the use of human-in-the-loop framework for minimizing the grading cost while guaranteeing the grading quality by allowing a SAS model to share the grading task with a human grader. Specifically, by introducing a confidence estimation method for indicating the reliability of the model predictions, one can guarantee the scoring quality by utilizing only predictions with high reliability for the scoring results and casting predictions with low reliability to human graders. In our experiments, we investigate the feasibility of the proposed framework using multiple confidence estimation methods and multiple SAS datasets. We find that our human-in-the-loop framework allows automatic scoring models and human graders to achieve the target scoring quality.