Explanation Beyond Intuition: A Testable Criterion for Inherent Explainability

TL;DR

This paper introduces a formal, testable criterion for inherent explainability in AI models using graph theory, enabling verification of explanations and aiding regulatory compliance.

Contribution

It proposes a novel, verifiable criterion for inherent explainability based on graph theory, bridging intuition and formal testing in XAI.

Findings

The criterion matches existing intuitions on explainability.

It explains why some models are inherently explainable or not.

Demonstrates PREDICT model's inherent explainability.

Abstract

Inherent explainability is the gold standard in Explainable Artificial Intelligence (XAI). However, there is not a consistent definition or test to demonstrate inherent explainability. Work to date either characterises explainability through metrics, or appeals to intuition - "we know it when we see it". We propose a globally applicable criterion for inherent explainability. The criterion uses graph theory for representing and decomposing models for structure-local explanation, and recomposing them into global explanations. We form the structure-local explanations as annotations, a verifiable hypothesis-evidence structure that allows for a range of explanatory methods to be used. This criterion matches existing intuitions on inherent explainability, and provides justifications why a large regression model may not be explainable but a sparse neural network could be. We differentiate explainable -- a model that allows for explanation -- and \textit{explained} -- one that has a verified explanation. Finally, we provide a full explanation of PREDICT -- a Cox proportional hazards model of cardiovascular disease risk, which is in active clinical use in New Zealand. It follows that PREDICT is inherently explainable. This work provides structure to formalise other work on explainability, and allows regulators a flexible but rigorous test that can be used in compliance frameworks.