Optimizing In-Context Demonstrations for LLM-based Automated Grading

TL;DR

This paper introduces GUIDE, a novel framework that optimizes exemplar selection and rationale generation for LLM-based grading, significantly improving accuracy especially on borderline cases by focusing on rubric boundaries.

Contribution

GUIDE reframes exemplar selection as a boundary-focused optimization, employing contrastive operators and discriminative rationales to enhance LLM grading accuracy.

Findings

Outperforms standard retrieval baselines across multiple datasets.

Shows robust improvements on borderline and boundary cases.

Enhances rubric adherence and grading reliability.

Abstract

Automated assessment of open-ended student responses is a critical capability for scaling personalized feedback in education. While large language models (LLMs) have shown promise in grading tasks via in-context learning (ICL), their reliability is heavily dependent on the selection of few-shot exemplars and the construction of high-quality rationales. Standard retrieval methods typically select examples based on semantic similarity, which often fails to capture subtle decision boundaries required for rubric adherence. Furthermore, manually crafting the expert rationales needed to guide these models can be a significant bottleneck. To address these limitations, we introduce GUIDE (Grading Using Iteratively Designed Exemplars), a framework that reframes exemplar selection and refinement in automated grading as a boundary-focused optimization problem. GUIDE operates on a continuous loop of selection and refinement, employing novel contrastive operators to identify "boundary pairs" that are semantically similar but possess different grades. We enhance exemplars by generating discriminative rationales that explicitly articulate why a response receives a specific score to the exclusion of adjacent grades. Extensive experiments across datasets in physics, chemistry, and pedagogical content knowledge demonstrate that GUIDE significantly outperforms standard retrieval baselines. By focusing the model's attention on the precise edges of rubric, our approach shows exceptionally robust gains on borderline cases and improved rubric adherence. GUIDE paves the way for trusted, scalable assessment systems that align closely with human pedagogical standards.