Scaling Down to Scale Up: A Cost-Benefit Analysis of Replacing OpenAI's LLM with Open Source SLMs in Production

TL;DR

This paper systematically evaluates open-source small language models (SLMs) as cost-effective and reliable alternatives to proprietary large language models (LLMs) like GPT-4 in real-world applications.

Contribution

It introduces SLaM, an open-source tool for comprehensive evaluation of SLMs, and provides a thorough comparison of SLMs versus GPT-4 in a product setting.

Findings

SLMs achieve 5x to 29x cost reduction compared to GPT-4.

SLMs show competitive quality and improved performance consistency.

Systematic evaluation methodology for SLMs in production environments.

Abstract

Many companies use large language models (LLMs) offered as a service, like OpenAI's GPT-4, to create AI-enabled product experiences. Along with the benefits of ease-of-use and shortened time-to-solution, this reliance on proprietary services has downsides in model control, performance reliability, uptime predictability, and cost. At the same time, a flurry of open-source small language models (SLMs) has been made available for commercial use. However, their readiness to replace existing capabilities remains unclear, and a systematic approach to holistically evaluate these SLMs is not readily available. This paper presents a systematic evaluation methodology and a characterization of modern open-source SLMs and their trade-offs when replacing proprietary LLMs for a real-world product feature. We have designed SLaM, an open-source automated analysis tool that enables the quantitative and qualitative testing of product features utilizing arbitrary SLMs. Using SLaM, we examine the quality and performance characteristics of modern SLMs relative to an existing customer-facing implementation using the OpenAI GPT-4 API. Across 9 SLMs and their 29 variants, we observe that SLMs provide competitive results, significant performance consistency improvements, and a cost reduction of 5x~29x when compared to GPT-4.