Some thoughts about benchmarks for NMR

TL;DR

This paper discusses the considerations and lessons learned for creating benchmark repositories in the NMR community to advance system design, inspired by other AI fields.

Contribution

It provides insights into the requirements and decision-making processes necessary for establishing effective NMR benchmarks.

Findings

Lessons from other AI communities inform benchmark development.

Key requirements for NMR benchmark repositories are identified.

Guidelines for building effective NMR benchmarks are proposed.

Abstract

The NMR community would like to build a repository of benchmarks to push forward the design of systems implementing NMR as it has been the case for many other areas in AI. There are a number of lessons which can be learned from the experience of other communi- ties. Here are a few thoughts about the requirements and choices to make before building such a repository.