ZEBRA: Leveraging Model-Behavioral Knowledge for Zero-Annotation Preference Dataset Construction

TL;DR

ZEBRA introduces a zero-annotation framework that constructs preference datasets for language model alignment by analyzing model behavior, eliminating the need for manual labeling and reducing costs while maintaining high performance.

Contribution

ZEBRA is the first to leverage model behavior knowledge for preference dataset construction without instance-level annotations, enabling scalable and cost-effective alignment data generation.

Findings

Achieves alignment performance comparable to supervised methods

Eliminates manual annotation costs

Demonstrates scalable preference data construction

Abstract

Recent efforts in LLM alignment have focused on constructing large-scale preference datasets via human or Artificial Intelligence (AI) annotators. However, such approaches rely on instance-wise supervision, incurring substantial annotation cost and limited interpretability. In this paper, we propose ZEBRA - a model behavior-wise zero-annotation framework that constructs preference data by leveraging model behavior knowledge derived from benchmark performances. ZEBRA binarizes response pairs by evaluating the quality and similarity of their origin models, entirely bypassing instance-level annotation. This allows scalable, controllable, and cost-effective alignment data generation. Empirical results show that ZEBRA achieves alignment performance comparable to instance-supervised methods, despite requiring no manual or model-based labeling.