Small But Funny: A Feedback-Driven Approach to Humor Distillation

TL;DR

This paper proposes a feedback-driven distillation method where large language models serve as both teachers and critics, significantly improving small language models' ability to generate humor by incorporating feedback during training.

Contribution

Introducing a dual-role approach for LLMs as teachers and critics to enhance humor generation in small language models through feedback-based distillation.

Findings

Feedback incorporation narrows the performance gap in humor generation.

Dual-role LLMs outperform imitation-only distillation methods.

Feedback-driven distillation improves complex language task transfer.

Abstract

The emergence of Large Language Models (LLMs) has brought to light promising language generation capabilities, particularly in performing tasks like complex reasoning and creative writing. Consequently, distillation through imitation of teacher responses has emerged as a popular technique to transfer knowledge from LLMs to more accessible, Small Language Models (SLMs). While this works well for simpler tasks, there is a substantial performance gap on tasks requiring intricate language comprehension and creativity, such as humor generation. We hypothesize that this gap may stem from the fact that creative tasks might be hard to learn by imitation alone and explore whether an approach, involving supplementary guidance from the teacher, could yield higher performance. To address this, we study the effect of assigning a dual role to the LLM - as a "teacher" generating data, as well as a "critic" evaluating the student's performance. Our experiments on humor generation reveal that the incorporation of feedback significantly narrows the performance gap between SLMs and their larger counterparts compared to merely relying on imitation. As a result, our research highlights the potential of using feedback as an additional dimension to data when transferring complex language abilities via distillation.