Call for establishing benchmark science and engineering

TL;DR

This paper advocates for the development of a unified benchmark science and engineering discipline to establish standardized benchmarks across multiple fields, addressing current fragmentation and proposing a new journal for dissemination.

Contribution

It introduces the concept of benchmark science and engineering, categorizes existing benchmarks, and proposes a hierarchical standardization framework and a dedicated journal for the field.

Findings

Identifies five categories of benchmarks in multi-disciplines.

Highlights the need for a meta-benchmark to evaluate benchmarks.

Proposes a multi-disciplinary benchmark hierarchy and a new journal.

Abstract

Currently, there is no consistent benchmarking across multi-disciplines. Even no previous work tries to relate different categories of benchmarks in multi-disciplines. This article investigates the origin and evolution of the benchmark term. Five categories of benchmarks are summarized, including measurement standards, standardized data sets with defined properties, representative workloads, representative data sets, and best practices, which widely exist in multi-disciplines. I believe there are two pressing challenges in growing this discipline: establishing consistent benchmarking across multi-disciplines and developing meta-benchmark to measure the benchmarks themselves. I propose establishing benchmark science and engineering; one of the primary goals is to set up a standard benchmark hierarchy across multi-disciplines. It is the right time to launch a multi-disciplinary benchmark, standard, and evaluation journal, TBench, to communicate the state-of-the-art and state-of-the-practice of benchmark science and engineering.