Concept Distillation from Strong to Weak Models via Hypotheses-to-Theories Prompting

TL;DR

Concept Distillation (CD) is an automatic prompt optimization method that improves weaker language models by leveraging strong models to generate and filter rules, significantly boosting performance on complex tasks like code generation and mathematical reasoning.

Contribution

This paper introduces Concept Distillation, a novel automated prompt enhancement technique that effectively transfers knowledge from strong to weak models for complex tasks.

Findings

Mistral-7B's accuracy on Multi-Arith increased by 20%.

Phi-3-mini-3.8B's accuracy on HumanEval rose by 34%.

CD outperforms other automated prompt optimization methods.

Abstract

Hand-crafting high quality prompts to optimize the performance of language models is a complicated and labor-intensive process. Furthermore, when migrating to newer, smaller, or weaker models (possibly due to latency or cost gains), prompts need to be updated to re-optimize the task performance. We propose Concept Distillation (CD), an automatic prompt optimization technique for enhancing weaker models on complex tasks. CD involves: (1) collecting mistakes made by weak models with a base prompt (initialization), (2) using a strong model to generate reasons for these mistakes and create rules/concepts for weak models (induction), and (3) filtering these rules based on validation set performance and integrating them into the base prompt (deduction/verification). We evaluated CD on NL2Code and mathematical reasoning tasks, observing significant performance boosts for small and weaker language models. Notably, Mistral-7B's accuracy on Multi-Arith increased by 20%, and Phi-3-mini-3.8B's accuracy on HumanEval rose by 34%. Compared to other automated methods, CD offers an effective, cost-efficient strategy for improving weak models' performance on complex tasks and enables seamless workload migration across different language models without compromising performance.