Predicting Disagreement with Human Raters in LLM-as-a-Judge Difficulty Assessment without Using Generation-Time Probability Signals

TL;DR

This paper introduces a novel method to predict when LLM-generated difficulty ratings will disagree with human raters, using embedding space analysis instead of generation-time probabilities, improving prediction accuracy.

Contribution

The proposed approach predicts rating disagreements without relying on generation-time signals, leveraging geometric consistency in embedding space for better disagreement detection.

Findings

Higher AUC achieved in predicting disagreement compared to probability-based baselines.

Effective on English CEFR-based sentence difficulty assessment datasets.

Applicable to multiple LLMs like GPT-OSS-120B and Qwen3-235B-A22B.

Abstract

Automatic generation of educational materials using large language models (LLMs) is becoming increasingly common, but assigning difficulty levels to such materials still requires substantial human effort. LLM-as-a-Judge has therefore attracted attention, yet disagreement with human raters remains a major challenge. We propose a method for predicting which LLM-generated difficulty ratings are likely to disagree with human raters, so that such cases can be sent for re-rating. Unlike prior approaches, our method does not rely on generation-time probability signals, which must be collected during rating generation and are often difficult to compare across LLMs. Instead, exploiting the fact that difficulty is an ordinal scale, we use a separate embedding space, such as ModernBERT, and identify disagreement candidates based on the geometric consistency of the rating set. Experiments on English CEFR-based sentence difficulty assessment with GPT-OSS-120B and Qwen3-235B-A22B showed that the proposed method achieved higher AUC for predicting disagreement with human raters than probability-based baselines.