Plug-in and Fine-tuning: Bridging the Gap between Small Language Models and Large Language Models

TL;DR

PiFi is a framework that enhances small language models by integrating a frozen layer from a large language model and fine-tuning, achieving high performance with low computational costs across various NLP tasks.

Contribution

This paper introduces PiFi, a novel method that combines a frozen LLM layer with an SLM, improving generalization and task performance efficiently.

Findings

PiFi improves performance on multiple NLP tasks.

PiFi enhances generalization to unseen domains.

PiFi leverages LLM knowledge effectively.

Abstract

Large language models (LLMs) are renowned for their extensive linguistic knowledge and strong generalization capabilities, but their high computational demands make them unsuitable for resource-constrained environments. In contrast, small language models (SLMs) are computationally efficient but often lack the broad generalization capacity of LLMs. To bridge this gap, we propose PiFi, a novel framework that combines the strengths of both LLMs and SLMs to achieve high performance while maintaining efficiency. PiFi integrates a single frozen layer from an LLM into a SLM and fine-tunes the combined model for specific tasks, boosting performance without a significant increase in computational cost. We show that PiFi delivers consistent performance improvements across a range of natural language processing tasks, including both natural language understanding and generation. Moreover, our findings demonstrate PiFi's ability to effectively leverage LLM knowledge, enhancing generalization to unseen domains and facilitating the transfer of linguistic abilities.