Efficient LLM Context Distillation

TL;DR

This paper evaluates context distillation as an efficient method for adapting large language models, demonstrating its comparable in-domain accuracy and better out-of-domain generalization than in-context learning, with lower data and computational requirements.

Contribution

It provides a comparative analysis showing context distillation's effectiveness and efficiency in model adaptation, especially for small datasets, relative to in-context learning and fine-tuning.

Findings

Context distillation achieves similar in-domain accuracy to ICL.

It outperforms ICL in out-of-domain generalization.

It requires less data and computation than fine-tuning.

Abstract

Large Language Models (LLMs) demonstrate proficiency across diverse tasks but often require targeted adaptations for specific applications. Various methods have been proposed to facilitate this adaptation, including fewshot fine-tuning, in-context learning, and context distillation. This paper specifically investigates context distillation a method that extends the utility of task-specific examples by internalizing them, thus augmenting the example set accessible for model inference. We conduct a comparative analysis of context distillation with in-context learning (ICL) and few-shot fine-tuning (FT), aiming to ascertain the efficacy of context distillation in adapting models using minimal in-context examples. Employing matched datasets from Mobach, our experiments leverage OPT models of various sizes. The results indicate that context distillation effectively adapts models, with student models attaining comparable in-domain and out-of-domain accuracies to in-context learning. Although context distillation surpasses ICL in out-of-domain generalization, it does not achieve the performance levels of FT. However, the reduced dataset size and computational demands position context distillation as a viable alternative, especially for smaller datasets. Overall, this study presents context distillation as an efficient and potent method for customizing LLMs to specific tasks.