A Scoresheet for Explainable AI

TL;DR

This paper introduces a scoresheet framework to specify and assess explainability requirements in AI systems, addressing the gap between high-level standards and practical needs across various applications.

Contribution

It develops a versatile scoresheet for evaluating explainability tailored to stakeholder needs, applicable to multiagent systems and other AI technologies.

Findings

Scoresheet effectively captures explainability requirements.

Application examples demonstrate its versatility.

Guidance improves practical adoption.

Abstract

Explainability is important for the transparency of autonomous and intelligent systems and for helping to support the development of appropriate levels of trust. There has been considerable work on developing approaches for explaining systems and there are standards that specify requirements for transparency. However, there is a gap: the standards are too high-level and do not adequately specify requirements for explainability. This paper develops a scoresheet that can be used to specify explainability requirements or to assess the explainability aspects provided for particular applications. The scoresheet is developed by considering the requirements of a range of stakeholders and is applicable to Multiagent Systems as well as other AI technologies. We also provide guidance for how to use the scoresheet and illustrate its generality and usefulness by applying it to a range of applications.