KNIFE: Distilling Reasoning Knowledge From Free-Text Rationales

TL;DR

KNIFE is a novel method that distills reasoning knowledge from free-text rationales into smaller language models, significantly improving their reasoning performance without requiring large models or direct rationale input.

Contribution

The paper introduces KNIFE, a technique that effectively transfers reasoning knowledge from large models with rationales into smaller models, enhancing reasoning capabilities.

Findings

KNIFE outperforms baselines on question-answering tasks.

FTR quality significantly impacts KNIFE's effectiveness.

Small models can learn reasoning skills from large models using KNIFE.

Abstract

Language models (LMs) have yielded impressive results on many language reasoning tasks, but their unexpected errors raise doubts about their reasoning abilities. In light of this, there is growing interest in finetuning/prompting LMs with both task instances and their associated free-text rationales (FTRs), which explain the correct reasoning process for predicting the correct task output (i.e., how to be "right for the right reasons"). However, existing finetuning methods fail to improve LM performance, while prompting needs prohibitively large (i.e., >50B) LMs to work well. We propose KNIFE, which shows that reasoning knowledge can be effectively distilled from FTRs into a small (i.e., <1B) LM and improve the LM's performance. First, KNIFE finetunes a teacher LM (given task input and FTR) to predict the task output, transferring reasoning knowledge from the FTRs to the teacher's hidden states. Second, KNIFE finetunes a student LM (given task input only) such that its hidden states are aligned with the teacher's. Thus, the student is endowed with reasoning knowledge but can be used for inference without direct FTR input. On two question-answering datasets, KNIFE outperforms various finetuning and prompting baselines in fully-supervised and low-resource settings. Also, we observe that FTR quality is crucial to KNIFE's performance.