EasyMath: A 0-shot Math Benchmark for SLMs

TL;DR

EasyMath is a compact benchmark designed to evaluate small language models' practical math reasoning abilities across diverse categories in a zero-shot setting, highlighting size and training effects.

Contribution

It introduces a new, comprehensive zero-shot math benchmark for small language models covering multiple categories and evaluates model performance with detailed checks.

Findings

Accuracy improves with model size and training

Chain-of-thought reasoning provides modest gains

Model consistency increases at larger scales

Abstract

EasyMath is a compact benchmark for practical math reasoning in small language models. It covers thirteen categories, from basic arithmetic and order of operations to word problems, algebraic expressions, edge cases, and omits specialist topics. We tested 23 models (14M to 4B parameters) using exact, numerical, and symbolic checks on free-form answers in a zero-shot setting. Accuracy rises with size and training, chain-of-thought adds modest gains, and consistency improves at scale.