Benchmarking Distilled Language Models: Performance and Efficiency in Resource-Constrained Settings

TL;DR

This paper benchmarks distilled language models, showing they are significantly more efficient and can match or surpass larger models in performance, making AI more accessible in resource-limited settings.

Contribution

It provides a comprehensive performance and efficiency comparison of distilled models versus vanilla and proprietary models, highlighting distillation's effectiveness.

Findings

Distilled 8B models are over 2,000 times more compute-efficient than vanilla models.

Distilled models achieve reasoning capabilities comparable to or better than much larger models.

Distillation significantly improves the performance-to-compute ratio of language models.

Abstract

Knowledge distillation offers a transformative pathway to developing powerful, yet efficient, small language models (SLMs) suitable for resource-constrained environments. In this paper, we benchmark the performance and computational cost of distilled models against their vanilla and proprietary counterparts, providing a quantitative analysis of their efficiency. Our results demonstrate that distillation creates a superior performance-tocompute curve. We find that creating a distilled 8B model is over 2,000 times more compute-efficient than training its vanilla counterpart, while achieving reasoning capabilities on par with, or even exceeding, standard models ten times its size. These findings validate distillation not just as a compression technique, but as a primary strategy for building state-of-the-art, accessible AI