Generating Data-Driven Reasoning Rubrics for Domain-Adaptive Reward Modeling

TL;DR

This paper introduces a data-driven method to create detailed reasoning error taxonomies, called rubrics, which improve error detection and reward modeling for LLMs in technical domains, reducing reliance on costly gold labels.

Contribution

It presents a novel approach to automatically generate granular reasoning rubrics that enhance error detection and reward modeling in domain-specific LLM applications.

Findings

Error classification with rubrics outperforms baselines in technical domains

Rewards based on rubrics improve task accuracy by +45%

Method reduces gold label requirements to 20% of traditional needs

Abstract

An impediment to using Large Language Models (LLMs) for reasoning output verification is that LLMs struggle to reliably identify errors in thinking traces, particularly in long outputs, domains requiring expert knowledge, and problems without verifiable rewards. We propose a data-driven approach to automatically construct highly granular reasoning error taxonomies to enhance LLM-driven error detection on unseen reasoning traces. Our findings indicate that classification approaches that leverage these error taxonomies, or "rubrics", demonstrate strong error identification compared to baseline methods in technical domains like coding, math, and chemical engineering. These rubrics can be used to build stronger LLM-as-judge reward functions for reasoning model training via reinforcement learning. Experimental results show that these rewards have the potential to improve models' task accuracy on difficult domains over models trained by general LLMs-as-judges by +45%, and approach performance of models trained by verifiable rewards while using as little as 20% as many gold labels. Through our approach, we extend the usage of reward rubrics from assessing qualitative model behavior to assessing quantitative model correctness on tasks typically learned via RLVR rewards. This extension opens the door for teaching models to solve complex technical problems without a full dataset of gold labels, which are often highly costly to procure.