SLM-Bench: A Comprehensive Benchmark of Small Language Models on Environmental Impacts--Extended Version

TL;DR

SLM-Bench introduces a comprehensive benchmark for small language models, evaluating their accuracy, efficiency, and environmental impact across multiple tasks and hardware setups, promoting fair comparison and reproducibility.

Contribution

This work presents the first holistic benchmark for SLMs that includes environmental and efficiency metrics, filling a gap in systematic evaluation tools.

Findings

Some models excel in accuracy, others in energy efficiency.

SLM-Bench enables fair comparison across diverse hardware configurations.

The benchmark promotes reproducibility with open-source tools.

Abstract

Small Language Models (SLMs) offer computational efficiency and accessibility, yet a systematic evaluation of their performance and environmental impact remains lacking. We introduce SLM-Bench, the first benchmark specifically designed to assess SLMs across multiple dimensions, including accuracy, computational efficiency, and sustainability metrics. SLM-Bench evaluates 15 SLMs on 9 NLP tasks using 23 datasets spanning 14 domains. The evaluation is conducted on 4 hardware configurations, providing a rigorous comparison of their effectiveness. Unlike prior benchmarks, SLM-Bench quantifies 11 metrics across correctness, computation, and consumption, enabling a holistic assessment of efficiency trade-offs. Our evaluation considers controlled hardware conditions, ensuring fair comparisons across models. We develop an open-source benchmarking pipeline with standardized evaluation protocols to facilitate reproducibility and further research. Our findings highlight the diverse trade-offs among SLMs, where some models excel in accuracy while others achieve superior energy efficiency. SLM-Bench sets a new standard for SLM evaluation, bridging the gap between resource efficiency and real-world applicability.