From Calculation to Adjudication: Examining LLM judges on Mathematical Reasoning Tasks

TL;DR

This paper investigates the effectiveness of large language models as judges for mathematical reasoning tasks, revealing their strengths in identifying better models but limitations in improving overall task performance.

Contribution

It provides a detailed analysis of LLM judges on mathematical tasks and demonstrates that simple features can predict their judgments with high accuracy.

Findings

Easy samples are easier to judge correctly.

Judgment performance correlates with model quality.

LLM judges often favor higher-quality models even when incorrect.

Abstract

To reduce the need for human annotations, large language models (LLMs) have been proposed as judges of the quality of other candidate models. The performance of LLM judges is typically evaluated by measuring the correlation with human judgments on generative tasks such as summarization or machine translation. In contrast, we study LLM judges on mathematical reasoning tasks. These tasks require multi-step reasoning, and the correctness of their solutions is verifiable, enabling a more objective evaluation. We perform a detailed performance analysis and find that easy samples are easy to judge, and difficult samples are difficult to judge. Our analysis uncovers a strong correlation between judgment performance and the candidate model task performance, indicating that judges tend to favor higher-quality models even if their answer is incorrect. As a consequence, we test whether we can predict the behavior of LLM judges using simple features such as part-of-speech tags and find that we can correctly predict 70%-75% of judgments. We conclude this study by analyzing practical use cases, showing that LLM judges consistently detect the on-average better model but largely fail if we use them to improve task performance.