Reinforcement Learning-based Knowledge Distillation with LLM-as-a-Judge

TL;DR

This paper introduces an RL framework for knowledge distillation that uses LLMs as judges to evaluate outputs over unlabeled data, eliminating the need for ground truth labels and improving reasoning capabilities.

Contribution

It proposes a novel RL-based approach utilizing LLM judges for label-free knowledge distillation, enhancing reasoning performance without relying on verifiable ground truth labels.

Findings

Achieves performance gains on math reasoning benchmarks.

Efficient reward computation with single-token judge outputs.

Effective training signals generated by LLM-based evaluators.

Abstract

Reinforcement Learning (RL) has been shown to substantially improve the reasoning capability of small and large language models (LLMs), but existing approaches typically rely on verifiable rewards, hence ground truth labels. We propose an RL framework that uses rewards from an LLM that acts as a judge evaluating model outputs over large amounts of unlabeled data, enabling label-free knowledge distillation and replacing the need of ground truth supervision. Notably, the judge operates with a single-token output, making reward computation efficient. When combined with verifiable rewards, our approach yields substantial performance gains across math reasoning benchmarks. These results suggest that LLM-based evaluators can produce effective training signals for RL fine-tuning.