What is the Role of Small Models in the LLM Era: A Survey

TL;DR

This survey explores the evolving role of small models in the era dominated by large language models, emphasizing their collaboration potential, resource efficiency, and practical significance.

Contribution

It systematically analyzes the relationship between large and small models, highlighting the importance of small models in practical applications and resource-constrained environments.

Findings

Small models are crucial for resource-limited settings.

Collaboration between small and large models enhances performance.

Small models offer practical benefits despite the prominence of LLMs.

Abstract

Large Language Models (LLMs) have made significant progress in advancing artificial general intelligence (AGI), leading to the development of increasingly large models such as GPT-4 and LLaMA-405B. However, scaling up model sizes results in exponentially higher computational costs and energy consumption, making these models impractical for academic researchers and businesses with limited resources. At the same time, Small Models (SMs) are frequently used in practical settings, although their significance is currently underestimated. This raises important questions about the role of small models in the era of LLMs, a topic that has received limited attention in prior research. In this work, we systematically examine the relationship between LLMs and SMs from two key perspectives: Collaboration and Competition. We hope this survey provides valuable insights for practitioners, fostering a deeper understanding of the contribution of small models and promoting more efficient use of computational resources. The code is available at https://github.com/tigerchen52/role_of_small_models