Evaluating Austrian A-Level German Essays with Large Language Models for Automated Essay Scoring

TL;DR

This study assesses the effectiveness of large language models in automatically grading Austrian A-level German essays, revealing current limitations in accuracy for real-world application.

Contribution

It evaluates multiple state-of-the-art open-weight LLMs for rubric-based essay scoring in German, highlighting their current performance and limitations.

Findings

Maximum 40.6% agreement with human raters on sub-dimensions

Only 32.8% of final grades matched human assessments

Smaller models are insufficient for reliable real-world grading

Abstract

Automated Essay Scoring (AES) has been explored for decades with the goal to support teachers by reducing grading workload and mitigating subjective biases. While early systems relied on handcrafted features and statistical models, recent advances in Large Language Models (LLMs) have made it possible to evaluate student writing with unprecedented flexibility. This paper investigates the application of state-of-the-art open-weight LLMs for the grading of Austrian A-level German texts, with a particular focus on rubric-based evaluation. A dataset of 101 anonymised student exams across three text types was processed and evaluated. Four LLMs, DeepSeek-R1 32b, Qwen3 30b, Mixtral 8x7b and LLama3.3 70b, were evaluated with different contexts and prompting strategies. The LLMs were able to reach a maximum of 40.6% agreement with the human rater in the rubric-provided sub-dimensions, and only 32.8% of final grades matched the ones given by a human expert. The results indicate that even though smaller models are able to use standardised rubrics for German essay grading, they are not accurate enough to be used in a real-world grading environment.