On the relationship between Benchmarking, Standards and Certification in Robotics and AI

TL;DR

This paper explores how benchmarking, standards, and certification are interconnected in robotics and AI, emphasizing their importance for responsible innovation and system validation.

Contribution

It clarifies the relationships among benchmarking, standards, and certification, highlighting their combined role in promoting responsible development in robotics and AI.

Findings

Standards provide normative requirements for AI and robotics systems.

Certification depends on conformance to standards for system approval.

Benchmarks serve as informal standards for system evaluation.

Abstract

Benchmarking, standards and certification are closely related processes. Standards can provide normative requirements that robotics and AI systems may or may not conform to. Certification generally relies upon conformance with one or more standards as the key determinant of granting a certificate to operate. And benchmarks are sets of standardised tests against which robots and AI systems can be measured. Benchmarks therefore can be thought of as informal standards. In this paper we will develop these themes with examples from benchmarking, standards and certification, and argue that these three linked processes are not only useful but vital to the broader practice of Responsible Innovation.