LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructions

TL;DR

This paper introduces LaMini-LM, a set of small, diverse models distilled from large instruction-tuned LLMs, achieving competitive performance across multiple NLP benchmarks with significantly reduced size.

Contribution

We develop a large, diverse instruction dataset and fine-tune a herd of small models, demonstrating effective knowledge distillation from large LLMs into compact models.

Findings

LaMini-LM models perform comparably to larger baselines.

The instruction dataset covers broad topics ensuring diversity.

Small models achieve high performance on multiple NLP tasks.

Abstract

Large language models (LLMs) with instruction fine-tuning demonstrate superior generative capabilities. However, these models are resource-intensive. To alleviate this issue, we explore distilling knowledge from instruction-tuned LLMs into much smaller ones. To this end, we carefully develop a large set of 2.58M instructions based on both existing and newly-generated instructions. In addition to being sizable, we design our instructions to cover a broad set of topics to ensure diversity. Extensive analysis of our instruction dataset confirms its diversity, and we generate responses for these instructions using gpt-3.5-turbo. Leveraging these instructions, we fine-tune a diverse herd of models, collectively referred to as LaMini-LM, which includes models from both the encoder-decoder and decoder-only families, with varying sizes. We evaluate the performance of our models using automatic metrics on 15 different natural language processing (NLP) benchmarks, as well as through human assessment. The results demonstrate that our proposed LaMini-LM models are comparable to competitive baselines, while being much smaller in size.