Need a Small Specialized Language Model? Plan Early!

TL;DR

This paper presents methods for efficiently creating specialized small language models from large pretraining sets, using importance sampling and a novel projected network architecture, to achieve good performance with limited data and budgets.

Contribution

It introduces importance sampling for domain-specific pretraining and a new projected network architecture for efficient model adaptation, advancing small model specialization techniques.

Findings

Importance sampling effectively mimics specialized data for small models.

Projected networks enable efficient adaptation of large models to small, specialized networks.

Both methods show empirical success across various domains and training constraints.

Abstract

Large language models are versatile tools but are not suitable for small inference budgets. Small models have more efficient inference, but their lower capacity means that their performance can be good only if one limits their scope to a specialized domain. This paper explores how to get good specialized small language models using a large, generic, pretraining set and a limited amount of specialized data. We consider two scenarios, depending on whether (i) one can afford pretraining a model for each specialization task, or (ii) one wants to cheaply adapt a single pretrained model for each task. In the first scenario, we propose an effective solution based on importance sampling: we resample the pretraining set to imitate the specialization data and train a small model on it. In the second scenario, we propose a novel architecture, projected networks (PN). PN is a large network whose parameters can be linearly projected into a small network for specialization. For both scenarios, we demonstrate the empirical effectiveness of our solutions across various domains, training set sizes, and training budgets.