Bridging the Gap in XAI-Why Reliable Metrics Matter for Explainability and Compliance

TL;DR

This paper emphasizes the importance of standardized, reliable metrics in explainable AI to enhance accountability, trustworthiness, and regulatory compliance, especially in high-stakes sectors.

Contribution

It introduces a Governance by Metrics paradigm, linking explainability evaluation with systemic AI accountability and regulatory standards.

Findings

Identifies limitations in current XAI metrics regarding faithfulness and tamper resistance.

Proposes a hierarchical model connecting transparency, tamper resistance, and legal alignment.

Outlines a roadmap for integrating explainability metrics into AI assurance pipelines.

Abstract

Reliable explainability is not only a technical goal but also a cornerstone of private AI governance. As AI models enter high-stakes sectors, private actors such as auditors, insurers, certification bodies, and procurement agencies require standardized evaluation metrics to assess trustworthiness. However, current XAI evaluation metrics remain fragmented and prone to manipulation, which undermines accountability and compliance. We argue that standardized metrics can function as governance primitives, embedding auditability and accountability within AI systems for effective private oversight. Building upon prior work in XAI benchmarking, we identify key limitations in ensuring faithfulness, tamper resistance, and regulatory alignment. Furthermore, interpretability can directly support model alignment by providing a verifiable means of ensuring behavioral integrity in General Purpose AI (GPAI) systems. This connection between interpretability and alignment positions XAI metrics as both technical and regulatory instruments that help prevent alignment faking, a growing concern among oversight bodies. We propose a Governance by Metrics paradigm that treats explainability evaluation as a central mechanism of private AI governance. Our framework introduces a hierarchical model linking transparency, tamper resistance, scalability, and legal alignment, extending evaluation from model introspection toward systemic accountability. Through conceptual synthesis and alignment with governance standards, we outline a roadmap for integrating explainability metrics into continuous AI assurance pipelines that serve both private oversight and regulatory needs.