A BenchCouncil View on Benchmarking Emerging and Future Computing

TL;DR

This paper advocates for establishing a formal benchmark science for emerging computing, proposing a unifying framework and methodology to address complex measurement challenges and support future technological developments.

Contribution

It introduces a comprehensive benchmark definition, a conceptual framework, and a traceable learning-based methodology to advance benchmark science and engineering for emerging computing.

Findings

Proposes a unifying benchmark definition and framework

Develops a traceable supervised learning-based benchmarking methodology

Lays foundation for benchmark science and engineering in emerging computing

Abstract

The measurable properties of the artifacts or objects in the computer, management, or finance disciplines are extrinsic, not inherent -- dependent on their problem definitions and solution instantiations. Only after the instantiation can the solutions to the problem be measured. The processes of definition, instantiation, and measurement are entangled, and they have complex mutual influences. Meanwhile, the technology inertia brings instantiation bias -- trapped into a subspace or even a point at a high-dimension solution space. These daunting challenges, which emerging computing aggravates, make metrology can not work for benchmark communities. It is pressing to establish independent benchmark science and engineering. This article presents a unifying benchmark definition, a conceptual framework, and a traceable and supervised learning-based benchmarking methodology, laying the foundation for benchmark science and engineering. I also discuss BenchCouncil's plans for emerging and future computing. The ongoing projects include defining the challenges of intelligence, instinct, quantum computers, Metaverse, planet-scale computers, and reformulating data centers, artificial intelligence for science, and CPU benchmark suites. Also, BenchCouncil will collaborate with ComputerCouncil on open-source computer systems for planet-scale computing, AI for science systems, and Metaverse.