WRAVAL -- WRiting Assist eVALuation

TL;DR

This paper introduces WRAVAL, a new evaluation framework that assesses small language models' effectiveness in practical, non-reasoning tasks, emphasizing their potential in industrial applications and edge computing scenarios.

Contribution

The paper presents a novel evaluation framework combining data generation, prompt-tuning, and LLM-based assessment to better measure SLMs' capabilities in real-world tasks.

Findings

SLMs perform well in non-reasoning industrial tasks

The framework highlights SLMs' potential with task-specific finetuning

Evaluation tools are provided for benchmarking SLMs and LLMs

Abstract

The emergence of Large Language Models (LLMs) has shifted language model evaluation toward reasoning and problem-solving tasks as measures of general intelligence. Small Language Models (SLMs) -- defined here as models under 10B parameters -- typically score 3-4 times lower than LLMs on these metrics. However, we demonstrate that these evaluations fail to capture SLMs' effectiveness in common industrial applications, such as tone modification tasks (e.g., funny, serious, professional). We propose an evaluation framework specifically designed to highlight SLMs' capabilities in non-reasoning tasks where predefined evaluation datasets don't exist. Our framework combines novel approaches in data generation, prompt-tuning, and LLM-based evaluation to demonstrate the potential of task-specific finetuning. This work provides practitioners with tools to effectively benchmark both SLMs and LLMs for practical applications, particularly in edge and private computing scenarios. Our implementation is available at: https://github.com/amazon-science/wraval.