Causal Distillation: Transferring Structured Explanations from Large to Compact Language Models

TL;DR

This paper presents a new method called Causal Distillation to transfer causal reasoning abilities from large models to smaller ones, using structured explanations and a novel coherence metric to improve and evaluate causal reasoning in open-source models.

Contribution

The paper introduces a framework for distilling causal explanations from large to small models and proposes the Causal Explanation Coherence metric for evaluating explanation quality.

Findings

Smaller models can learn causal reasoning through structured explanation distillation.

The CEC metric effectively measures the coherence and faithfulness of generated causal explanations.

The framework improves the causal reasoning capabilities of open-source language models.

Abstract

Large proprietary language models exhibit strong causal reasoning abilities that smaller open-source models struggle to replicate. We introduce a novel framework for distilling causal explanations that transfers causal reasoning skills from a powerful teacher model to a compact open-source model. The key idea is to train the smaller model to develop causal reasoning abilities by generating structured cause-and-effect explanations consistent with those of the teacher model. To evaluate the quality of the student-generated explanations, we introduce a new metric called Causal Explanation Coherence (CEC) to assess the structural and logical consistency of causal reasoning. This metric uses sentence-level semantic alignment to measure how well each part of the generated explanation corresponds to the teacher's reference, capturing both faithfulness and coverage of the underlying causal chain. Our framework and the CEC metric provide a principled foundation for training smaller models to perform robust causal reasoning and for systematically assessing the coherence of explanations in language model outputs.