Few-Shot Knowledge Distillation of LLMs With Counterfactual Explanations

TL;DR

This paper introduces CoD, a novel few-shot knowledge distillation method that uses counterfactual explanations to efficiently transfer knowledge from large models to smaller ones with minimal data.

Contribution

The paper proposes a new distillation strategy leveraging counterfactual explanations to improve knowledge transfer in few-shot settings, with theoretical and empirical validation.

Findings

CoD outperforms standard methods with fewer samples.

Using CFEs enhances the decision boundary approximation.

Method is effective across various datasets and LLMs.

Abstract

Knowledge distillation is a promising approach to transfer capabilities from complex teacher models to smaller, resource-efficient student models that can be deployed easily, particularly in task-aware scenarios. However, existing methods of task-aware distillation typically require substantial quantities of data which may be unavailable or expensive to obtain in many practical scenarios. In this paper, we address this challenge by introducing a novel strategy called Counterfactual-explanation-infused Distillation CoD for few-shot task-aware knowledge distillation by systematically infusing counterfactual explanations. Counterfactual explanations (CFEs) refer to inputs that can flip the output prediction of the teacher model with minimum perturbation. Our strategy CoD leverages these CFEs to precisely map the teacher's decision boundary with significantly fewer samples. We provide theoretical guarantees for motivating the role of CFEs in distillation, from both statistical and geometric perspectives. We mathematically show that CFEs can improve parameter estimation by providing more informative examples near the teacher's decision boundary. We also derive geometric insights on how CFEs effectively act as knowledge probes, helping the students mimic the teacher's decision boundaries more effectively than standard data. We perform experiments across various datasets and LLMs to show that CoD outperforms standard distillation approaches in few-shot regimes (as low as 8-512 samples). Notably, CoD only uses half of the original samples used by the baselines, paired with their corresponding CFEs and still improves performance.