Toward Trustworthy Difficulty Assessments: Large Language Models as Judges in Programming and Synthetic Tasks

TL;DR

This paper compares GPT-4o and LightGBM in assessing the difficulty of programming problems, revealing significant limitations of GPT-4o and emphasizing the importance of explicit features for trustworthy difficulty evaluation.

Contribution

It provides a systematic comparison between LLM-based and feature-based difficulty assessment methods, highlighting the shortcomings of GPT-4o and proposing insights for improving LLM trustworthiness in judging tasks.

Findings

LightGBM achieves 86% accuracy in difficulty classification.

GPT-4o reaches only 37.75% accuracy and overlooks numeric cues.

GPT-4o often mislabels synthetic Hard problems as Medium.

Abstract

Large Language Models (LLMs) have demonstrated impressive capabilities in natural language and code generation, and are increasingly deployed as automatic judges of model outputs and learning activities. Yet, their behavior on structured tasks such as predicting the difficulty of competitive programming problems remains under-explored. We conduct a systematic comparison of GPT-4o, used purely as a natural-language difficulty assessor, against an interpretable Light-GBM ensemble trained on explicit numeric and textual features. On a dataset of 1,825 LeetCode problems labeled Easy, Medium, or Hard, LightGBM attains 86% accuracy, whereas GPT-4o reaches only 37.75%. Detailed analyses, including confusion matrices and SHAP-based interpretability, show that numeric constraints -- such as input size limits and acceptance rates -- play a crucial role in separating Hard problems from easier ones. By contrast, GPT-4o often overlooks these cues and exhibits a strong bias toward simpler categories. We further probe GPT-4o through a synthetic Hard-problem generation protocol. Surprisingly, GPT-4o labels almost all of its own synthetic Hard problems as Medium, contradicting its tendency to downgrade real Hard problems to Easy. Our findings connect to recent work on LLMs-as-judges and automatic difficulty estimation in programming and education, and highlight concrete failure modes that must be addressed before LLM-based judges can be considered trustworthy in competitive programming, educational platforms, or reinforcement-learning pipelines.