CITING: Large Language Models Create Curriculum for Instruction Tuning

TL;DR

CITING introduces a novel approach where large language models generate curricula for instruction tuning, replacing human effort and significantly improving model performance across multiple datasets.

Contribution

The paper presents a new method where a teacher LLM creates rubrics and guides self-correction in student LLMs, enhancing instruction tuning without human-crafted datasets.

Findings

Achieves an average winning rate of 79.4% over SFT.

Demonstrates strong improvement in answer quality.

Outperforms several state-of-the-art baselines.

Abstract

The recent advancement of large language models (LLMs) has been achieved through a combo of instruction tuning and human alignment. However, building manually crafted instruction datasets and performing human alignment become the bottleneck for scaling the development of LLMs. In this paper, we exploit the idea of leveraging AI models in lieu of humans as the teacher to train student LLMs. Our method is inspired by how human students refine their writing skills by following the rubrics and learning from the revisions offered by their tutors. Specifically, we employ a teacher LLM to create a curriculum for instruction tuning of the student LLM, namely Curriculum Instruction TunING (CITING). It encompasses two main steps: (1) the teacher LLM crafts the rubrics for evaluating the answers corresponding to various types of questions, and (2) the student LLM learns to follow the rubrics and perform self-correction from the revision made by the teacher. We further iteratively carry out it to embody the procedure of CITING. We compare CITING to a series of state-of-the-art baselines on four datasets. Our method demonstrates strong improvement in terms of articulate, in-depth, and comprehensive by GPT-4 evaluation. Specifically, it achieves an average winning rate of 79.4% over SFT, 73.4% over RLHF, 78.1% over RRHF, and 76.3% over RAFT, respectively.