TaP: A Taxonomy-Guided Framework for Automated and Scalable Preference Data Generation

TL;DR

TaP is a framework that automates and scales the creation of multilingual preference datasets for large language models using a taxonomy-guided approach, improving model alignment and performance.

Contribution

It introduces a taxonomy-guided method for automated preference data generation, enabling scalable, diverse, and high-quality datasets across languages for LLM fine-tuning.

Findings

LLMs trained on TaP datasets outperform those trained on existing datasets.

Models trained on TaP data outperform larger open-source datasets.

TaP enables effective multilingual preference data generation.

Abstract

Conducting supervised and preference fine-tuning of large language models (LLMs) requires high-quality datasets to improve their ability to follow instructions and align with human preferences and values. However, constructing such datasets is resource-intensive, and most publicly available datasets are in English. To address these challenges, we propose the \underline{\textbf{Ta}}xonomy-Guided \underline{\textbf{P}}reference Data Generation (TaP) framework for automated, scalable preference dataset construction across languages. TaP uses a structured taxonomy to provide fine-grained control over dataset composition, ensuring diversity and broad coverage. We use TaP-generated datasets to perform supervised and preference fine-tuning on multiple LLMs. Experimental results demonstrate that LLMs trained on TaP-generated datasets outperform those trained on existing open-source datasets. Remarkably, LLMs trained on TaP-generated datasets outperform models trained on an open-source dataset that is 180$\times$ larger.