What Should Explanations Contain? A Human-Centered Explanation Content Model for Local, Post-Hoc Explanations

TL;DR

This paper develops a comprehensive human-centered content model for local, post-hoc explanations in industrial AI, based on qualitative analysis and expert validation.

Contribution

It introduces a 14-code explanation content model organized into four groups, validated for content adequacy and coding reliability.

Findings

Identified 14 explanation content codes relevant for industrial AI explanations.

Validated the content model with expert review and high inter-coder reliability.

Established a foundation for eliciting and evaluating explanation content in industrial AI systems.

Abstract

Which categories of explanation content are relevant for users of industrial AI systems, and how can those categories be organized for local, post-hoc explanations? To address these questions, a hybrid inductive-deductive qualitative content analysis was applied to 325 meaning units drawn from six user studies in building technology, manufacturing, AI software development, and hospital cybersecurity. The inductive phase produced an initial twelve-code structure. A theory-informed coverage assessment and expert review then added two further codes, Rule base and What-if backward, that were not instantiated in the corpus but correspond to system architectures documented in the XAI literature. The resulting fourteen-code model is organized into four groups: rule-based, causal, epistemic (actual), and epistemic (similar), with twelve codes grounded in the corpus and two as theoretical extensions. An eleven-member expert panel supported the content adequacy of all codes (I-CVI $\geq$ 0.82; scale-level agreement of 0.93 for relevance, 0.92 for boundary clarity, and 0.94 for understandability). A stratified subsample of 82 units (25\% of the corpus), coded independently by two researchers using the finalized codebook, yielded Krippendorff's $\alpha = 0.920$ and Cohen's $\kappa = 0.920$. The paper therefore establishes content adequacy and coding reproducibility for a content-level explanation model intended to support elicitation, specification, and later evaluation of explanation content in industrial AI systems. Behavioral validation of downstream effects remains future work.