Rubric-based On-policy Distillation

TL;DR

ROPD introduces a rubric-based on-policy distillation method that replaces teacher logits with semantic rubrics, enabling scalable, black-box model alignment with significant efficiency gains.

Contribution

It presents ROPD, a novel framework that derives prompt-specific rubrics from teacher-student contrasts for effective on-policy distillation without needing teacher logits.

Findings

ROPD outperforms logit-based OPD in most scenarios.

Achieves up to 10x sample efficiency improvement.

Provides a scalable, black-box-compatible distillation baseline.

Abstract

On-policy distillation (OPD) is a powerful paradigm for model alignment, yet its reliance on teacher logits restricts its application to white-box scenarios. We contend that structured semantic rubrics can serve as a scalable alternative to teacher logits, enabling OPD using only teacher-generated responses. To prove it, we introduce ROPD, a simple yet foundational framework for rubric-based OPD. Specifically, ROPD induces prompt-specific rubrics from teacher-student contrasts, and then utilizes these rubrics to score the student rollouts for on-policy optimization. Empirically, ROPD outperforms the advanced logit-based OPD methods across most scenarios, and achieving up to a 10x gain in sample efficiency. These results position rubric-based OPD as a flexible, black-box-compatible alternative to the prevailing logit-based OPD, offering a simple yet strong baseline for scalable distillation across proprietary and open-source LLMs. Code is available at https://github.com/Peregrine123/ROPD_official.