Are Small Language Models Ready to Compete with Large Language Models for Practical Applications?

TL;DR

This paper introduces a framework for evaluating small open language models in practical applications, demonstrating that with proper selection, they can outperform some large models like GPT-4o in certain tasks.

Contribution

The work proposes a novel evaluation framework for small LMs across various practical aspects and compares 10 models to identify optimal choices for specific applications.

Findings

Small LMs can outperform SOTA large models with proper selection.

The framework effectively measures semantic correctness across different tasks and domains.

Certain small LMs match or surpass GPT-4o performance in specific scenarios.

Abstract

The rapid rise of Language Models (LMs) has expanded their use in several applications. Yet, due to constraints of model size, associated cost, or proprietary restrictions, utilizing state-of-the-art (SOTA) LLMs is not always feasible. With open, smaller LMs emerging, more applications can leverage their capabilities, but selecting the right LM can be challenging as smaller LMs do not perform well universally. This work tries to bridge this gap by proposing a framework to experimentally evaluate small, open LMs in practical settings through measuring semantic correctness of outputs across three practical aspects: task types, application domains, and reasoning types, using diverse prompt styles. It also conducts an in-depth comparison of 10 small, open LMs to identify the best LM and prompt style depending on specific application requirements using the proposed framework. We also show that if selected appropriately, they can outperform SOTA LLMs like DeepSeek-v2, GPT-4o, GPT-4o-mini, Gemini-1.5-Pro, and even compete with GPT-4o.