Learning from Committee: Reasoning Distillation from a Mixture of Teachers with Peer-Review

TL;DR

This paper introduces a novel knowledge distillation method called FAIR that uses peer-review and mistake explanations from multiple teachers to improve small language models' reasoning abilities across various tasks.

Contribution

The paper proposes a new peer-review based distillation approach that enhances instruction data quality by focusing on mistake explanations and selective rationales from multiple teachers.

Findings

Improves reasoning performance on mathematical, commonsense, and logical tasks.

Reduces flawed rationale influence through peer-review filtering.

Enhances small models' reasoning capabilities effectively.

Abstract

While reasoning capabilities typically emerge in large language models (LLMs) with tens of billions of parameters, recent research focuses on improving smaller open-source models through knowledge distillation (KD) from commercial LLMs. However, many of these studies rely solely on responses from a single LLM as the gold rationale, unlike the natural human learning process, which involves understanding both the correct answers and the reasons behind mistakes. In this paper, we introduce a novel Fault-Aware DistIllation via Peer-Review (FAIR) approach: 1) instead of merely obtaining rationales from teachers, our method asks teachers to identify and explain the student's mistakes, providing customized instruction learning data; 2) we design a simulated peer-review process between teacher LLMs, and selects only the generated rationales above the acceptance threshold, which reduces the chance of teachers guessing correctly with flawed rationale, improving instructional data quality. Comprehensive experiments and analysis on mathematical, commonsense, and logical reasoning tasks demonstrate the effectiveness of our method. Our code is available at https://github.com/zhuochunli/Learn-from-Committee.