Benchmarking TinyML Systems: Challenges and Direction

TL;DR

This paper discusses the importance of benchmarking TinyML systems, reviews current challenges, and proposes four benchmarks to facilitate fair comparison and progress in ultra-low-power machine learning hardware.

Contribution

It introduces four new benchmarks for TinyML systems and outlines a methodology for their selection, advancing the standardization efforts in the field.

Findings

Identification of key challenges in TinyML benchmarking

Proposal of four specific benchmarks for TinyML hardware

Insights from the TinyMLPerf working group on benchmarking standards

Abstract

Recent advancements in ultra-low-power machine learning (TinyML) hardware promises to unlock an entirely new class of smart applications. However, continued progress is limited by the lack of a widely accepted benchmark for these systems. Benchmarking allows us to measure and thereby systematically compare, evaluate, and improve the performance of systems and is therefore fundamental to a field reaching maturity. In this position paper, we present the current landscape of TinyML and discuss the challenges and direction towards developing a fair and useful hardware benchmark for TinyML workloads. Furthermore, we present our four benchmarks and discuss our selection methodology. Our viewpoints reflect the collective thoughts of the TinyMLPerf working group that is comprised of over 30 organizations.