Focusing on Students, not Machines: Grounded Question Generation and Automated Answer Grading

TL;DR

This paper develops a system for generating grounded questions and automatically grading student answers from class materials, demonstrating that large language models can effectively perform automated grading with increasing accuracy as model size grows.

Contribution

It introduces a novel document chunking method for PDFs, a new benchmark for short answer grading, and evaluates LLMs' ability to automate grading with minimal human oversight.

Findings

High-quality question and answer generation from study materials

Large language models improve grading accuracy with increased size

Current systems still require human oversight in exam settings

Abstract

Digital technologies are increasingly used in education to reduce the workload of teachers and students. However, creating open-ended study or examination questions and grading their answers is still a tedious task. This thesis presents the foundation for a system that generates questions grounded in class materials and automatically grades student answers. It introduces a sophisticated method for chunking documents with a visual layout, specifically targeting PDF documents. This method enhances the accuracy of downstream tasks, including Retrieval Augmented Generation (RAG). Our thesis demonstrates that high-quality questions and reference answers can be generated from study material. Further, it introduces a new benchmark for automated grading of short answers to facilitate comparison of automated grading systems. An evaluation of various grading systems is conducted and indicates that Large Language Models (LLMs) can generalise to the task of automated grading of short answers from their pre-training tasks. As with other tasks, increasing the parameter size of the LLMs leads to greater performance. Currently, available systems still need human oversight, especially in examination scenarios.