A framework for causal concept-based model explanations

TL;DR

This paper introduces a framework for explaining AI models using causal concepts, ensuring explanations are both understandable and accurate.

Contribution

The novelty lies in a causal concept-based framework for XAI that ensures explanations are faithful and interpretable.

Findings

Explanations are generated using the probability of sufficiency of concept interventions.

A proof-of-concept model demonstrates the framework on the CelebA dataset.

The framework emphasizes the importance of aligning explanation interpretation with generation context.

Abstract

This work presents a conceptual framework for causal concept-based post-hoc explainable artificial intelligence (XAI), based on the requirements that explanations for non-interpretable models must be both understandable and faithful to the model being explained. Local and global explanations are generated by calculating the probability of sufficiency of concept interventions. Example explanations are presented, generated with a proof-of-concept model made to explain classifiers trained on the CelebA dataset. Understandability is demonstrated through a clear concept-based vocabulary, subject to an implicit causal interpretation. Fidelity is addressed by highlighting important framework assumptions, stressing that the context of explanation interpretation must align with the context of explanation generation.

Figures

The graph details the complete causal framework over variables u∪z∪w∪x∪{y^} that is presented in this section. Variables x and y^ are included with gray background, as these are observed variables from the point of view of h, the model to be explained. Variables z, w, u are generally unobserved. Variables u are not circled in the graph, to emphasize the distinction between the exogenous u representing influence on z originating outside of the model, and the remaining, endogenous variables.

The figure details an example causal graph over concept variables z1, z2, z3, z4, z5, each with an exogenous parent ui.

The figure details the complete causal flow of the framework when u, z, and causal model M are as shown in Figure 2. The distinction between ground truth y and prediction ŷ discussed above is illustrated here by considering a model h that is learned to predict the value of concept z1, such that y = z1. In this case, while x is a cause of the prediction y^, the ground truth y is a cause of x.

The causal model over concepts relevant for the age-classifier.

The causal model over concepts relevant for the attractiveness-classifier.

Example images to be explained together with several counterfactual variations. The left-most images are the originals, with z-vectors (Gender = 1, Glasses = 0, Makeup = 0, Gray Hair = 0, Young = 1) (top) and (Gender = 0, Glasses = 0, Makeup = 1, Gray Hair = 0, Young = 1) (bottom). From second from the left to right, the columns contain the counterfactuals with Glasses = 1, Gray Hair = 1, Young = 0, and Gray Hair = 1 & Young = 0, all else as in original. The images are classified as young, young, old, old, old (top), and young, young, young, old, old (bottom). Thus, all images are correctly classified except the top third from the left, which is wrongly classified as old after intervening on gray hair only. Note that the original images shown in this figure and in the rest of the figures presented in this section are generated with the explanation generator, by inputting dataset images along with their original z-vector. This is done to ensure copyright compliance. The real dataset images can be found through the CelebA website (https://mmlab.ie.cuhk.edu.hk/projects/CelebA.html).

Counterfactual versions of the image to the left (classification results are given in parentheses): Young = 0 (h(x′) is young), Young = 0 & Gray Hair = 1 (Old), Young = 0 & Glasses = 1 (Old), Young = 0 & Gray Hair = 1 & Glasses = 1 (Old). The z-vector of the original image is (Gender = 0, Glasses = 0, Makeup = 1, Gray Hair = 0, Young = 1). Note that the original images shown in this figure and in the rest of the figures presented in this section are generated with the explanation generator, by inputting dataset images along with their original z-vector. This is done to ensure copyright compliance. The real dataset images can be found through the CelebA website (https://mmlab.ie.cuhk.edu.hk/projects/CelebA.html).

The results of feature attribution by Grad-CAM on the original images in Figure 6. Note that the original images shown in this figure and in the rest of the figures presented in this section are generated with the explanation generator, by inputting dataset images along with their original z-vector. This is done to ensure copyright compliance. The real dataset images can be found through the CelebA website (https://mmlab.ie.cuhk.edu.hk/projects/CelebA.html).

The figure shows (from left to right) the original image from Figure 7, Grad-CAM explanation of why the subject was classified as young, counterfactual image after do(Young = 0), Grad-CAM explanation of the erroneous classification that the subject is young. Note that the original images shown in this figure and in the rest of the figures presented in this section are generated with the explanation generator, by inputting dataset images along with their original z-vector. This is done to ensure copyright compliance. The real dataset images can be found through the CelebA website (https://mmlab.ie.cuhk.edu.hk/projects/CelebA.html).

Tables

Table 1. Probability of sufficiency for counterfactual intervention on the variable z¯=Young for the leftmost image in Figure 7.

Model type	p(y^do(Young=0)=0|x=, ŷ=1)
FSCM	0.121
PSCM (canonical)	[0.121, 0.138]
Independence	0.0

Table 2. Probability of sufficiency for singleton interventions (i.e., |z¯|=1).

Intervention	p(y^do(z¯=z¯)=0|zs,ŷ=1)
Gray Hair = 1	0.318
Glasses = 1	0.002
Makeup = 0	0.000
Young = 0	0.972

Table 3. Probability of sufficiency for Gray Hair, broken down on each subject's Gender and status regarding Glasses.

Subgroup			p(y^do(z¯=z¯)=0|zs,ŷ=1)
Gender	Glasses	Count
Male	Both	519	0.580
Male	True	43	0.814
Male	False	476	0.559
Female	Both	986	0.181
Female	True	15	0.400
Female	False	971	0.177
Both	Both	1505	0.318

Table 4. Probability of sufficiency for counterfactual intervention on the variable z¯=Young.

Model type	p(y^do(Young=0)=0|Young=1,y^=1)
FSCM	0.972
PSCM (canonical)	[0.970, 0.973]
Independence	0.966

Table 5. Probability of sufficiency for singleton interventions to leaf variables in the explanation graph in Figure 5, conditioned on Young = 1 and Gender, for changing the output of the attractiveness classifier from not attractive to attractive.

Intervention		p(y^do(z¯=z¯)=1|z¯=¬z¯,Young=1,Gender,ŷ=0)
z¯	z¯	Gender = 0	Gender = 1
5 o'Clock shadow	1	0.139	0.117
	0	-	0.000
Arched eyebrows	1	0.034	0.003
	0	0.085	0.214
Bushy eyebrows	1	0.177	0.109
	0	0.000	0.000
Big nose	1	0.000	0.004
	0	0.574	0.147
Pointy nose	1	0.350	0.306
	0	0.000	0.000
Smiling	1	0.242	0.181
	0	0.000	0.009

Table 6. Probability of sufficiency for counterfactual intervention do(Makeup = 1) for the subgroup Young = 1, Gender = 0 and Makeup = 0 classified as not attractive.

Model type	p(y^do(Makeup=1)=1|Young=1,Gender=0,Makeup=0,y^=0)
FSCM	0.410
PSCM (canonical)	[0.401,0.442]
Independence	0.353

Table 7. TCAV scores quantifying the importance of various concepts for classifying attractiveness.

Intervention		TCAVℓ(ŷ=1,z¯,Dŷ=0∩z¯=¬z¯∩Gender)
z¯	z¯	Gender = 0	Gender = 1
5 o'clock shadow	1	0.256	0.251
	0	-	0.788
Arched eyebrows	1	0.995	0.977
	0	0.011	0.011
Bushy eyebrows	1	0.182	0.292
	0	0.835	0.796
Big nose	1	0.314	0.270
	0	0.745	0.714
Pointy nose	1	0.660	0.646
	0	0.325	0.344
Smiling	1	0.642	0.588
	0	0.408	0.443

Equations

Full text

Introduction

1

With state-of-the-art machine learning models used for an increasing variety of high stakes decision making, a well-documented challenge is the lack of transparency introduced by the complexity of the models employed (Goodman and Flaxman, 2017; Gerlings et al., 2021). The field of explainable artificial intelligence (XAI) aims to ensure models are equipped with the ability to provide explanations for their predictions. One way to approach the task of explaining model behavior is to design methods that generate explanations for a given pre-trained, non-interpretable model. This is referred to as post-hoc XAI (Barredo Arrieta et al., 2020), and an obvious advantage of this approach is that a system's ability to explain its behavior is treated independently of optimizing model performance.

XAI explanations are expected to meet certain criteria. It is critical that (1) explanations are understandable to the explainee (Miller, 2019), and that (2) explanations are faithful to the explanandum, i.e., the model (Jacovi and Goldberg, 2020). Understandability is receiver-dependent, but generally requires meaningful content and a concise presentation. Fidelity is evaluated with respect to the model to be explained, and any explanation must accurately reflect the model's behavior. This includes no presence of bias, which for instance could be the result of strategically restricting the scope of explanation.

While understandability and fidelity are the independent qualities of explanations, maximizing each in isolation is not trivial. For instance, a challenge to be considered is that explanations that align with what is expected may be perceived as more understandable. This introduces a preference for explanations that confirm prior beliefs, which results in bias if allowed to influence explanation selection. Moreover, a faithful, unbiased explanation that is misinterpreted will equally mislead the explainee.

This work presents a causal post-hoc XAI framework for explaining arbitrary models, aiming to facilitate both understandability and fidelity within a defined context.

First, understandability is addressed by arguing that explanations should be causal and expressed in an interpretable vocabulary. The requirement for explanations to be causal is based on this being argued the implicit structure of explanation employed in human communication (Miller, 2019). Thus, ensuring causally valid explanations aids in aligning intended interpretation with actual interpretation. This, in turn, eliminates the need for additional instructions on how to interpret the explanation, allowing a more succinct representation.

The requirement for understandable vocabulary emphasizes the likely discrepancy between the set of data features that the model takes as input, and the set of features ideal for explanation. This distinction is the key foundation of a group of XAI methods collectively known as concept-based XAI (C-XAI) (Poeta et al., 2025). The central assumption of this approach is that while models are equally capable of processing information in the form of pixels, graphs, and tokens, human communication typically assumes a vocabulary consisting of units corresponding to high-level, semantically meaningful concepts.

Second, fidelity is addressed through identifying and minimizing bias introduced by the explanation generation process. Assumptions made prior to generation of explanations are discussed, to provide a well-defined context for explanation. This includes details regarding the comprehensiveness of the vocabulary and the choice of causal model, and how the scope of explanation in turn is influenced.

The final framework admits causal reasoning about model behavior, including generation of counterfactual events as contrastive explanations. Local explanations are generated counterfactually, relating change to probability of sufficiency. Global explanations are generated both with counterfactuals and with interventional queries.

The contributions of the study are the following:

A theoretical XAI framework for post-hoc, causal explanation generation is presented (Section 4.1)Both concept attributions and counterfactually sound contrastive concept-based explanations are defined based on counterfactual calculation of the probability of sufficiency (Section 4.2)A proof-of-concept post-hoc explanation model is detailed and applied to generate local and global explanations for classifiers trained on the celebA dataset (Section 5).

Background

2

Notation

2.1

Notation x/x denotes a scalar/set of random variable(s), with element i of x denoted x_i. Notation x/x is used to denote a scalar/set of value(s). The domain of a variable x is denoted by Ω_x.

Causality

2.2

Relevant background for discussing causal modeling and inference is reviewed here.

Structural causal models

2.2.1

Given a set of variables v describing a domain, with causal structure represented by graph $[eqn]$ . Counterfactual inference in v requires a structural causal model (SCM) (Galles and Pearl, 1997; Pearl, 2009). An SCM has components $[eqn]$ , where v is a set of endogenous variables, u is a set of exogenous variables, and $[eqn]$ is a set of deterministic structural functions, one for each endogenous variable. A structural function fv(pav) = v determines the value of variable v given its parents pav in the causal graph $[eqn]$ . The exogenous variables u are introduced as root variables in $[eqn]$ , such that each u is a parent to exactly one endogenous variable v. Here, causal sufficiency is assumed (u_i⫫uj_ for all u_i, uj_ ∈ u, i ≠ j). Such SCMs are called Markovian SCMs (Avin et al., 2005) and do not allow exogenous causal confounding. Note that an endogenous variable v may be modeled without an exogenous parent if v is assumed to be deterministically given by its endogenous parents.

The distinction between endogenous and exogenous variables is here assumed to be made based on semantic interpretation. All variables in the environment described by $[eqn]$ that represent the concepts of known interpretation are modeled as endogenous variables, regardless of whether it is observed or not. Then, the exogenous variables are included where relevant to account for causal influence of origin outside of the model scope. Exogenous variables are thus always unobserved.

The SCM is further rendered probabilistic by introducing a probability distribution P(u) over the exogenous variables. An SCM where all components $[eqn]$ are defined, is referred to as a fully specified SCM (FSCM) (Pearl, 2009; Zaffalon et al., 2024). If the probability distribution P(u) is unknown, the SCM is referred to as a partially specified SCM (PSCM). The most general SCM lets each structural function in $[eqn]$ admit every possible function from its endogenous inputs to its output variable and is referred to as canonical.

Counterfactual queries

2.2.2

A counterfactual query combines observed evidence with a hypothetical, retrospective intervention (Pearl, 2009). An intervention, denoted do(v = v), assigns a value for a variable while breaking the dependence between the intervened variable and its causal parents, to simulate action, as opposed to change in observation. A counterfactual query is of the form $[eqn]$ 1, read as the probability that variable w had taken value w′ if variable v had been intervened to take value v′, given that the observed event is v = v, w = w.

Probability of sufficiency

2.2.3

Given two binary variables, v and w, the Probability of Sufficiency (PS) is given as P(w_do(v = 1)_ = 1|w = 0, v = 0) (Pearl, 2009), and gives the probability to which variable v intervened to take on value 1 is sufficient for variable w to take on value 1, given that it is observed that v = 0 and w = 0.

For binary variables of certain interpretation, the query may be read as the probability of variable v being a sufficient cause of variable w, implicitly referencing values at 1 as activations. Generalizing the probability of sufficiency to all discrete variables as $[eqn]$ with v ≠ v′ and w ≠ w′, the query is rather read with explicit reference to the values, i.e., the probability of v = v being a sufficient cause of w = w in a context where neither v nor w is observed.

Post-hoc XAI

2.3

This section reviews relevant topics concerning post-hoc XAI.

Type of explanation

2.3.1

Several types of explanations are defined within the field of XAI. These include feature attributions (Ribeiro et al., 2016; Lundberg and Lee, 2017), which are the scores assigned for each data feature to quantify the contribution of the feature toward the model's outcome. By a generalized interpretation, this type of explanation extends to concept attributions, where attribution is measured for variables not explicit in the data representation.

Another type of explanation is the contrastive explanation (Wachter et al., 2018). For a given sample x to be explained, contrasting data samples that are similar to x but that differ in model prediction are presented as the explanation. This type of explanation is also known as a counterfactual explanation in the XAI field, but the term counterfactual in this context has a different meaning than it does in causality. To avoid confusion, contrastive is used whenever it is the XAI type of explanation that is referred to, while counterfactual explanation refers to explanations generated with counterfactual queries as detailed in Section 2.2.

Scope of explanation

2.3.2

The scope of explanations for XAI range from local to global (Angelov et al., 2021). Local explanations consider a single data sample x and explain the model's prediction for this sample by considering the model's behavior in a local neighborhood around x only. Global explanations instead explain the model by identifying global trends representative of universal model behavior. Global explanation techniques can be applied to a subgroup of data samples with certain attributes in common, or to infer relationships descriptive of all data samples.

Goal of explanation

2.3.3

A definition of XAI given by DARPA refers to XAI as “AI systems that can explain their rationale to a human user, characterize their strengths and weaknesses, and convey an understanding of how they will behave in the future” (Gunning and Aha, 2019). In line with this, post-hoc XAI aims to ensure explainability for non-interpretable models, by employing techniques that reveal why the model predicts a certain output. This in turn can permit better evaluation of model performance, increase user trust in the decision making process, or help users understand how a decision can be changed in situations where decisions are not permanent. The latter requires a special case of explanation, where the explanation vocabulary is restricted according to actionability, i.e., whether a variable may be feasibly intervened upon in a current world state. Explaining under actionability constraints with the goal of changing a model decision is referred to as algorithmic recourse. While it requires reliable causal modeling to accurately predict the result of the suggested interventions, it is concerned with the future outcome of actualized interventions only.

The type of post-hoc explanation discussed in this study neither makes assumptions regarding the explainee guiding the process of explanation, nor about the performance or deployment stage of the model. This is done to ensure that explanations are faithful to the model, also when the model cannot be assumed to behave as intended or expected. Counterfactual reasoning about retrospective interventions on all variables is permitted, with no requirement for actionability. While the framework discussed may be applied for algorithmic recourse with some adjustments, the details of this is out of scope of this study.

Concept vocabulary

2.3.4

Concept-based explanations use concepts rather than data features as their vocabulary. While there exist multiple definitions of concepts for XAI, it is the type of concept identified as symbolic concepts by Poeta et al. (2025), that is referred to here. Thus, a concept corresponds to a high-level attribute or characteristic of the data, with semantic meaning as defined by a human.

Because concepts are not explicitly defined in the data features, it is common to use probing to detect neurons, or combinations of neurons, that best represent target concepts. Such probes are often linear, but may also be non-linear in cases where the concept is not linearly separable. Furthermore, probes can be trained in both supervised and unsupervised manners, where the former requires labeled data of the human-specified concept, and the latter identifies concepts that are important to the model, but not necessarily interpretable to humans.

To assess the importance of identified concepts, concept attribution methods such as TCAV (Kim et al., 2018) quantify the importance of a concept z, probed in layer ℓ, for a given output class y. Concepts are learned using linear classifier probes such as logistic regression, resulting in the coefficient-vector vz, which is often interpreted as the direction in which the concept is represented. Let fℓ(x) be the result of forwarding the input x to the intermediate activation in layer ℓ, and gℓ, y(·) be the remainder of the network, mapping those activations to the logit of class y. Kim et al. (2018) define concept sensitivity as the directional derivative of gℓ, y wrt. fℓ(x) compared to the concept direction vz. For an input x, the sensitivity score is thus $[eqn]$ . These scores are aggregated over a background dataset, here a subset of the test data $[eqn]$ , providing the ratio of positive concept sensitivities as

$$[eqn]$$

where I[A] is 1 if A is true and 0 otherwise.

Related work

3

Several related works discuss post-hoc model explanation with causal models over concepts that are distinct from data features. Parafita and Vitrià (2019) use distributional causal graphs for casual modeling, with MLPs outputting concept distributions as structural functions. The framework does not allow exact counterfactual inference, and average effect of interventions is rather considered. The study further includes a conceptual discussion of requirements for a causal generator mapping concepts to data space. The methods presented by Dash et al. (2022) and Komanduri et al. (2024) restrict the process of learning structural functions for an FSCM to observation-based estimation of parameters of a function under model assumptions such as additive noise, Gaussian noise, and linearity. The counterfactual context u ~ p(u|x) given a data sample x is sampled via encodings u = q(x) learned from observations. The causal generator used by Dash et al. (2022) is based on a generative adversarial network (GAN) architecture, while Komanduri et al. (2024) employ a diffusion network.

The present work extends the modeling of structural equations beyond observation-based estimation under general assumptions. Various levels of modeling are considered, ranging from FSCMs designed with clear interpretation under assumptions based on domain knowledge to canonical PSCMs containing all FSCMs consistent with the data. This illustrates both the importance of modeling choices made regarding structural functions, and the potential for partial information in the form counterfactual intervals when complete FSCMs are not available.

Galhotra et al. (2021) present an XAI method based on calculating probabilities of sufficiency and necessity with FSCMs. Focusing on tabular data, they introduce causal modeling on top of features in data space, assuming these represent high-level concepts, with the option to include further concepts not part of the model input. While only applicable when features are meaningful concepts, this method does not require a causal generator. The method further assumes a preference regarding model output to be explained, such that explanations are framed in terms of which subsets of input values are sufficient to change a model prediction to a more preferable one, or which values are necessary to retain a preferable prediction whenever the outcome of the explanandum is preferred unaltered.

Moreover, Galhotra et al. (2021) include a discussion on bounding probabilities analytically whenever a fully specified SCM is not available. Counterfactual intervals for XAI is further explored in the present work, considering PSCMs for calculating intervals for counterfactual queries. This allows for well-defined subgroups of FSCMs to be considered, whenever imperfect domain knowledge is available, potentially reducing intervals compared to analytic calculations.

Probabilities of sufficiency and necessity are also considered for evaluation of XAI methods. Mothilal et al. (2021) define the ideal model explanation based on necessity and sufficiency, such that the explanation is a minimal subset of features that is both necessary and sufficient for a model outcome in a given context. With the assumption that the set of variables for explanation is the input features only, and that these are all independent, the types of XAI explanations are discussed in comparison. Probability of necessity is related to contrastive explanations by arguing the features changed a necessary cause for original model output. Sufficiency is related to feature attribution explanations, in that highly contributing features across contexts will correspond to sufficient features, more likely to cause the output regardless of other feature values. Experiment highlights high variation across methods in practice. Pawar et al. (2024), also assuming independent data features, similarly discuss sufficiency and necessity as the key criteria for existing XAI methods to be evaluated against, and extends this to cover different approaches to neighborhood generation as basis for explanation.

Beckers (2022) considers the related settings where the model to be explained predicts the outcome of actions, and thus is assumed to be causally meaningful. Both sufficient explanations and counterfactual explanations are defined for this scenario, under the assumption that the model always predict the correct action outcome. The types of explanations are further divided by distinguishing between degrees of causal influence, to identify actual causes for explanation.

Method

4

Causal XAI framework overview

4.1

This section details a framework for generation of causal post-hoc explanations for a model h in a meaningful vocabulary. The framework was first introduced by Bjøru et al. (2025b) as a means to classify various XAI methods according to their level of causal abstraction. Here, this framework is extended and actualized as a distinct XAI method for explaining black-box models.

The framework is based on the distinction between the explanation vocabulary, which is a set of meaningful concepts, and the data features that are the input of the model to be explained. In total, three model components complete the framework: (1) The black-box model h that will be explained, (2) a mapping α (with inverse α^−1^) from the explanation vocabulary concepts to the data representation as input to h, and (3) a causal model M over the explanation concepts. The framework includes the following sets of variables: z denotes the explanation concept vocabulary, u denotes the exogenous variables associated with z by M, x denotes the data features that are the input to the model h, $[eqn]$ denotes the output of h, and finally w denotes any noise or unknown factors of variation in x such that x is deterministically given by x = α(z, w).

Figure 1 summarizes this framework. The complete model with components M, α, h can be considered a single causal model over the set of variables $[eqn]$ , allowing functions α and h to be interpreted as representing causal mechanisms. The remainder of this section details each of the model components h, α, and M and their respective input and output.

The graph details the complete causal framework over variables u∪z∪w∪x∪{y^} that is presented in this section. Variables x and y^ are included with gray background, as these are observed variables from the point of view of h, the model to be explained. Variables z, w, u are generally unobserved. Variables u are not circled in the graph, to emphasize the distinction between the exogenous u representing influence on z originating outside of the model, and the remaining, endogenous variables.

The model h to be explained

4.1.1

The purpose of the causal framework is to allow explanation of a black-box model $[eqn]$ . This model h takes a set of input features x and predicts a corresponding output $[eqn]$ . No assumptions are made about the architecture of h, and importantly, no assumptions are made about the performance of h.

For h to be interpreted as a causal process, it is important to clearly distinguish prediction $[eqn]$ from the attribute y being predicted. Given an instance x, y = y is considered to represent the ground truth attribute corresponding to x, which generally may not be observed outside of training data. The model prediction $[eqn]$ is the model's guess as to what value y takes, such that it is not guaranteed that y = ŷ. This is clearly illustrated if for ground truth y that takes one of two values, h is chosen to be a constant function predicting the same ŷ for all inputs. Even if h perfectly predicts y by $[eqn]$ , the causal nature of these variables is distinct. While the prediction $[eqn]$ is always caused by x through h, it is rarely meaningful to model x as a cause of the true attribute y. Rather, y can be thought of as a concept that is part of the concept representation z with causal influence on x, or otherwise caused by concepts in z, and thus correlated with x by confounding. This distinction is illustrated further in an example presented in Section 5.

Explaining h is approached by relating change in $[eqn]$ to systematic change in x. However, the features x themselves will not in general constitute a useful vocabulary for explanation, as atomic features in x may correspond to pixels in an image, words in a text, nodes in a graph etc. Thus, to quantify meaningful change in x, a concept-based vocabulary with known causal structure is considered.

The mapping α from concepts to data features

4.1.2

The explanation vocabulary is denoted z, with each variable z_i_ representing a meaningful, high-level concept relevant to describe the information represented in x. Now z permits explanations in a language understandable for a human explainee.

To describe change in x by change in z, an invertible mapping α between the two sets of variables is required. Ideally, z is sufficient to generate x through a deterministic α, such that all information in x originates from meaningful concepts in z available for explanation. As this is rarely feasible in practice, another set of variables w is introduced, such that α(z, w) = x, with w encoding any unknown factors of variation in x not represented in z. Hence, α is kept a deterministic function. The variables w are not required to be interpretable, and it is assumed that the variable sets z and w are independent.

The independence z⫫w is assumed in order for the effect of interventions in z on x to be correctly modeled, which is not feasible if p(w|do(z)) ≠ p(w). It is noted that w is never to be considered part of the explanation language. For local explanations, w will be considered a context fixed at some value w = w, such that the effect of changes to z may be consistently mapped to x through α.

Another important assumption is thus that z, while not a complete model of all information present in x, constitutes a sufficiently comprehensive explanation language for h's decision logic to be (partially) exposed.

Conceptually, α is considered a decoder α(z, w) = x, and α^−1^ is a corresponding encoder α^−1^(x) = z, w. The independence z⫫w is assumed preserved by α^−1^ ° α. Dependence among variables in w is irrelevant such that independence here may be enforced if convenient. However, the variables of z can be dependent.

Note that while α is modeled as an invertible mapping, it is interpreted as representing a causal process in the data generative direction z, w → x. The causal influence of a world state on a data instance through the ground truth data generative process is modeled as causal influence of the concepts z as a high level representation of this world state, as well as unknown factors in w, on x.

The causal model M for causal reasoning

4.1.3

Expressing explanations in terms of causally meaningful change to z further requires a reliable model of z's causal structure, which represents the underlying physical mechanisms governing the relationships of these concepts in the real world. This structure is represented by a causal graph $[eqn]$ .

An explicit causal model M is introduced over variables z with structure $[eqn]$ . For maximum capacity for reasoning, M is considered an FSCM, and a final set of variables u is included, to act as M's exogenous variables. The model M is assumed to be Markovian as described in Section 2.2. Figure 2 presents an example graph $[eqn]$ over variable set $[eqn]$ , with Markovian exogenous variables $[eqn]$ .

The figure details an example causal graph over concept variables z1, z2, z3, z4, z5, each with an exogenous parent ui.

The requirement that M be an FSCM can be challenging to meet in practice, as no amount of data, observational or experimental, will reveal the exact nature of the underlying process to be captured by a structural function. Thus in-depth domain knowledge is typically required for the design of reasonable structural functions, with no automated way of verifying the reliability of a given model. Regardless, an FSCM is required for exact calculation of counterfactual queries, and so is the ideal modeling approach to M for causal explanations.

If a single, reasonable choice of FSCM cannot be provided, alternatives may be considered, at some cost to explanation quality. Assuming model complexity is sufficiently low, a PSCM may be considered, for which counterfactual queries may be bounded by intervals. The most general PSCM to be considered is referred to as canonical, which provides intervals guaranteed to contain the true counterfactual probability.

Finally, if no SCM can be provided, M can be modeled as a causal Bayesian network (CBN). In this setting, counterfactual reasoning is not feasible, and the framework may be applied for the calculation of interventional queries only.

With M, α, h as detailed, the causal flow is now summarized. The concept variables z have internal causal structure and are caused by their respective parents among the other concepts, as well as exogenous parents u, exemplified in Figure 2. The concepts z together with unknown factors w are the causes of the data features x through the data generative process represented by α. Finally, the data features x are the causes of the model prediction $[eqn]$ through the model h. Figure 3 summarizes the complete causal flow for the example in Figure 2. The concepts in z are thus indirect causes of prediction $[eqn]$ , and this is the relationship explored for explanation. While all variables are part of the extended causal model for explanation, only the variables in z are considered for intervention to detect causal relationship. Any intervention to variables x or $[eqn]$ would remove the link with concept space and the vocabulary chosen for explanation. As u and w are unobserved and uninterpretable, interventions here are not informative.

The figure details the complete causal flow of the framework when u, z, and causal model M are as shown in Figure 2. The distinction between ground truth y and prediction ŷ discussed above is illustrated here by considering a model h that is learned to predict the value of concept z1, such that y = z1. In this case, while x is a cause of the prediction y^, the ground truth y is a cause of x.

Causal XAI explanations

4.2

This section details how explanations are generated with the presented framework. The main operation of this generation is the calculation of probabilities for outcomes of interventions to subsets of concepts in z, in a given context.

Causal calculations

4.2.1

The framework yields the complete set of variables $[eqn]$ . The joint distribution over these variables factorizes as follows:

$$[eqn]$$

To explain h, queries of the generalized form $[eqn]$ are considered, quantifying the effect on the model prediction when intervening on a subset of concepts $[eqn]$ , conditioned on some observation o. The observation o is the context of the explanation, and different types of explanations are defined based on varying o.

Notation $[eqn]$ is used to denote the set of variables in z influenced by an intervention to $[eqn]$ but not in $[eqn]$ , i.e., the causal descendants of variables $[eqn]$ . Notation $[eqn]$ denotes variables in z unaffected by an intervention to $[eqn]$ , such that $[eqn]$ .

Local explanations

4.2.1.1

Local explanations are centered at a single data point x, ŷ = h(x), such that o = x, ŷ. Having that h is deterministic and that z, w = α^−1^(x), this is equivalent to o = z, w. The query becomes $[eqn]$ , which is a counterfactual query, calculated as follows:

$$[eqn]$$

Global explanations

4.2.1.2

Global explanations are either intervention-based or counterfactual, depending on the explanation context. In the most general case, o = ∅, and the explanation query simplifies as $[eqn]$ . This query is interventional, and does not require an SCM. Assuming the joint distribution $[eqn]$ is available, the query is calculated accordingly.

If the context of explanation is a subgroup, the query may become counterfactual. Generally, the context of explanation is o = zs, ŷ, where ŷ is a common prediction by h and zs is a subset of concept values, and the query becomes $[eqn]$ .

$$[eqn]$$

The query is estimated using an empirical distribution for p(x|zs, ŷ), with $[eqn]$ a dataset of samples consistent with the condition ŷ, zs. $[eqn]$ is calculated as detailed for local explanations.

If the subgroup is not conditioned on a common prediction, i.e., o = zs, the explanation query will be counterfactual or interventional depending on whether the intervention $[eqn]$ influences any of the context variables or not, respectively.

Probability of sufficiency for XAI

4.2.2

The general probability of sufficiency presented in Section 2.2 is adapted and applied for XAI explanations. The target variable of interest is the model prediction $[eqn]$ . Hypothetical change in the form of a counterfactual intervention is considered for a subset of concept variables $[eqn]$ . The context is either a local instance x, ŷ, z, w, or a subgroup defined by common concepts and prediction zs, ŷ such that $[eqn]$ . Then the probability of sufficiency for XAI is given by $[eqn]$ (local) or $[eqn]$ (subgroup), with ŷ′ ≠ ŷ and $[eqn]$ . The query quantifies the effect of the intervention $[eqn]$ on $[eqn]$ , relative to observed explanandum ŷ and $[eqn]$ .

Note that it is the counterfactual outcome that is the target of the sufficiency query, relative to the observation. The query is thus primarily explanatory with regards to the influence of the changed concepts toward the changed outcome. However, information about the causal influence of the observed concepts on the observed outcome is (partially) revealed, by letting sufficiency of change indicate necessity of the original value. Note that if $[eqn]$ , a new value $[eqn]$ being sufficient for change to occur in $[eqn]$ is not always equivalent to the observed value $[eqn]$ being necessary for the observed ŷ. Still, the respective scenarios are related.

Concept attributions

4.2.3

Given a single data point or a subgroup, the probability of sufficiency can be seen as a measure of attribution for the set of intervened features. The probability of sufficiency is to be read as a measure of attribution of intervened value $[eqn]$ toward alternative outcome ŷ′ in a context (e.g. x, ŷ), and is only indirectly descriptive of observed $[eqn]$ 's attribution toward observed ŷ. In the special case where $[eqn]$ , i.e., it is a single binary concept that is intervened, the probability of sufficiency can be interpreted as a concept importance score as defined for XAI, reflecting the probabilities for the sufficiency of the counterfactual concept value toward the counterfactual outcome and the necessity of the original concept value toward the original outcome.

Contrastive explanations

4.2.4

The presented framework enables the generation of contrastive explanations. The contrastive explanation form considered, based on the explanation form first introduced by Galhotra et al. (2021), is summarized as: For a data instance x that has concept values z, and for which a model h made the decision ŷ, the decision would have been ŷ′ with probability p had the concept(s) $[eqn]$ taken value(s) $[eqn]$ with $[eqn]$ the subset of the concepts that are changed by (hypothetical) intervention. The probability p is the local probability of sufficiency $[eqn]$ .

When selecting the best contrastive explanations from a set of contrastive samples, similarity with the original data instance is typically measured (Wachter et al., 2018). When data features are changed independently, similarity can be calculated as number of changed features. With the contrastive form presented here, similarity is interpreted as the number of variables intervened upon, i.e. $[eqn]$ . Additionally, the probability p must be considered when evaluating a potential explanation. Generally, it is reasonable to expect p to be high in order for the explanation to be useful. In some cases, a limit p^^ for this probability may be given, such that it is required that p ≥ p^^ for a sample to be considered an explanation. This in turn may allow a simplified explanation form where p is omitted, which is argued helpful in increasing understandability (Miller, 2019).

Searching for optimal contrastive explanations of the form presented, maximizing similarity and probability, is left for future work.

Examples

5

In this section, we will exemplify the explanatory process outlined above using an example. We work with the CelebA dataset (Liu et al., 2015). This dataset contains more than 200.000 images of celebrities' faces, each annotated with binary attributes describing facial features and accessories (e.g., if the subject is smiling, wearing glasses, of young age or is looking attractive). We will consider two tasks: classifying whether an image is of a Young person and whether the person is Attractive.

Models

5.1

We implemented the conceptual model in Figure 1 as a computational pipeline and will now describe each part separately. Each image $[eqn]$ in the dataset is annotated with a corresponding set of binary descriptors zi, and we will use a subset of the concepts in z as vocabulary when explaining the classifier h(·). For each explanation task, we chose the subset by investigating which concepts correlate the most with the target concept (Young and Attractive). The resulting subsets can be identified as the node-sets of the models in Figures 4, 5, respectively.

The causal model over concepts relevant for the age-classifier.

The causal model over concepts relevant for the attractiveness-classifier.

The causal model M

5.1.1

We build an FSCM over a selected subset of the concepts z. The causal structure was learned using an expert in the loop-method from pgmpy (Ankan and Textor, 2024), with a large language model (LLM) acting as the expert deciding on causal direction, and with the forced constraint that concepts Young and Gender should be independent. Conditional independence is also enforced for variables Smiling and Big Nose. The resulting structures are given in Figure 4 for the age classification and Figure 5 for the classification of attractiveness. The parameters of the FSCM defining the probability distribution p(u) were chosen to be sparse and meaningful, while compatible with the observed data $[eqn]$ ; see Appendix A for details. PSCM intervals are retrieved using the DCCC method for exact interval calculation (Bjøru et al., 2025a). Counterfactual queries are calculated using BCAUSE (Cabañas, 2025).

The classifier h

5.1.2

We use a 50-layer ResNet following the procedure of He et al. (2016). While our framework can explain the behavior of any classifier, irrespectively of merit, we note that both classifiers performed at an acceptable level, with accuracies above 80%.

The observation model α and its inverse α−1

5.1.3

To have a complete computational pipeline for counterfactual explanations, we train a generative model as a means to represent the mapping between concept-space and data-space. In particular, we provide counterfactual data examples as follows: First, the mapping α^−1^(x) = z, w is partially provided in the dataset itself, as each image xi is annotated with concepts zi. w is intended to capture the remaining information not described by z, e.g., image background. Here we simply set w = x, and will discuss the effects of this simplification in Section 5.3. We aim to evaluate a counterfactual explanation $[eqn]$ by calculating $[eqn]$ , cf. Section 4.2. This requires generation of counterfactual data-objects α(z′, w) where $[eqn]$ is consistent with the causal model M (see Equation 1). Our implementation adopts the StarGAN architecture (Choi et al., 2018) for image-to-image translation in order to provide the functionality of α(z′, w). StarGAN takes an image x and a binary vector of high-level concepts z′ as inputs and produces an alternation of x consistent with concept description $[eqn]$ but otherwise incentivized to be close to x.

Results

5.2

Local explanation

5.2.1

We first consider the images in Figure 6. For each row, the original image is given to the left, and we look for explanations of why these people are classified as being young. The explanation engine outlined above provides justification by answering counterfactual questions of the type “Would the model h(·) classify the person as young if 〈counterfactual〉?”, where the counterfactual is defined using the vocabulary shown in Figure 4. An intermediate step of producing explanations is to generate counterfactual images as described above. The counterfactual images shown in columns 2 and 3 are related to the concepts Glasses and Gray Hair. Adding glasses is not sufficient for the classifier to change its evaluation of either person in this example, but changing hair color is sufficient for the male to be classified as old2.

Example images to be explained together with several counterfactual variations. The left-most images are the originals, with z-vectors (Gender = 1, Glasses = 0, Makeup = 0, Gray Hair = 0, Young = 1) (top) and (Gender = 0, Glasses = 0, Makeup = 1, Gray Hair = 0, Young = 1) (bottom). From second from the left to right, the columns contain the counterfactuals with Glasses = 1, Gray Hair = 1, Young = 0, and Gray Hair = 1 & Young = 0, all else as in original. The images are classified as young, young, old, old, old (top), and young, young, young, old, old (bottom). Thus, all images are correctly classified except the top third from the left, which is wrongly classified as old after intervening on gray hair only. Note that the original images shown in this figure and in the rest of the figures presented in this section are generated with the explanation generator, by inputting dataset images along with their original z-vector. This is done to ensure copyright compliance. The real dataset images can be found through the CelebA website (https://mmlab.ie.cuhk.edu.hk/projects/CelebA.html).

Note that even though Young is a concept in the causal model, its value is in general distinct from the predicted value ŷ = h(x) as the classifier is not infallible. Intervening in the causal model M with do(Young = 0) will imply higher probabilities of both Gray Hair and Glasses (consider the causal model in Figure 4), in addition to other visual cues related to this concept, like higher affinity for wrinkles and under-eye bags. To examine counterfactuals with $[eqn]$ , we need to marginalize out the uncertainty in the concepts $[eqn]$ , cf. Equation 1. Two of the counterfactual data-points produced by α to make this calculation are shown in columns 4 and 5 of Figure 6. We eventually find that $[eqn]$ for both subjects, validating the explanation that the model h(·) would indeed consider both persons to look old if they were old.

We next turn to the images in Figure 7. Following from the original image on the left, the figure shows different counterfactual images all consistent with the intervention do(Young = 0). The first counterfactual image, obtained by the intervention do(Young = 0, Glasses = 0, Gray Hair = 0), is erroneously classified as being of a young person, while the other counterfactual images are correctly identified as being of old subjects. Using these counterfactual images to marginalize out the uncertainty over $[eqn]$ , we can calculate $[eqn]$ , $[eqn]$ , the probability that the classifier would classify the subject as old in the counterfactual world where Young = 0. The probabilistic model-class employed to do the calculations will provide different results, as showcased in Table 1.

Counterfactual versions of the image to the left (classification results are given in parentheses): Young = 0 (h(x′) is young), Young = 0 & Gray Hair = 1 (Old), Young = 0 & Glasses = 1 (Old), Young = 0 & Gray Hair = 1 & Glasses = 1 (Old). The z-vector of the original image is (Gender = 0, Glasses = 0, Makeup = 1, Gray Hair = 0, Young = 1). Note that the original images shown in this figure and in the rest of the figures presented in this section are generated with the explanation generator, by inputting dataset images along with their original z-vector. This is done to ensure copyright compliance. The real dataset images can be found through the CelebA website (https://mmlab.ie.cuhk.edu.hk/projects/CelebA.html).

While XAI systems that solely rely on visual assessment can be misleading (Adebayo et al., 2018), saliency maps are still an often-used technique for attribution-based explanations of image classification systems. We, therefore, compare the results of our causal concept-based explanations with the results of Grad-CAM (Selvaraju et al., 2017) using the implementation by Gildenblat (2021). Consider first the results in Figure 8 which are intended to explain why the two actual (far left) images in Figure 6 were classified as being young. Surprisingly, the top of the head is important for the female's classification while the area around the eyes and one of the ears are of most importance for the classification of the male. Next, Figure 9 investigates the classification of the subject in Figure 7. The first image was correctly classified as young, but according to the Grad-CAM explanation, the classifier has based this decision on elements detected in the image background. The explanation of the erroneously classified image to the right focuses on the forehead and the chin. Grad-CAM provides non-causal explanations, and therefore fails at giving counterfactual information. Instead of considering change, the explanation rather focuses on what part of the input space that are most indicative about the class being chosen the classifier. Furthermore, since the explanations are pixel-based, we do not get insights regarding why the highlighted areas are considered important.

The results of feature attribution by Grad-CAM on the original images in Figure 6. Note that the original images shown in this figure and in the rest of the figures presented in this section are generated with the explanation generator, by inputting dataset images along with their original z-vector. This is done to ensure copyright compliance. The real dataset images can be found through the CelebA website (https://mmlab.ie.cuhk.edu.hk/projects/CelebA.html).

The figure shows (from left to right) the original image from Figure 7, Grad-CAM explanation of why the subject was classified as young, counterfactual image after do(Young = 0), Grad-CAM explanation of the erroneous classification that the subject is young. Note that the original images shown in this figure and in the rest of the figures presented in this section are generated with the explanation generator, by inputting dataset images along with their original z-vector. This is done to ensure copyright compliance. The real dataset images can be found through the CelebA website (https://mmlab.ie.cuhk.edu.hk/projects/CelebA.html).

Global explanation

5.2.2

As an example of subgroup-explanations, we provide the probability that changing a single concept would suffice to change h(·)'s prediction of a person from being young ( $[eqn]$ ) to being old in Table 2. We can write this probability compactly as $[eqn]$ , cf. Equation 2, where the conditioning on $[eqn]$ is used to ensure that the value of z is indeed changed by the intervention. For simplicity we restrict ourselves to the most relevant singleton interventions, $[eqn]$ . As an example, consider the effect of setting $[eqn]$ , zs = 0 and $[eqn]$ . Now we are doing an investigation of the subpopulation of subjects that are classified as being young but do not have gray hair, looking for the counterfactual probability that the classifier would consider the subject to be old if their hair color had been changed to gray. The relatively large value for this probability shows that hair color is an important concept to explain the behavior of the classifier, which may be used to spark further investigation into potential bias in the classifier before it can be deployed. We investigate this further in Table 3, where the same calculation is broken down by gender and whether or not the subjects wear glasses. The effect is stronger among males than females, and albeit with a limited dataset, it also seems like the effect is stronger among those wearing glasses.

Since setting $[eqn]$ implies $[eqn]$ , we must again marginalize out the uncertainty over $[eqn]$ . How this is done depends on how the causal model M is represented (cf. Table 4). For this particular query, the canonical interval is narrow, such that the choice of FSCM has limited influence on the calculated probability of sufficiency. Using a model with independent concept variables slightly underestimates the global effect of the intervention, as was also shown to be the case locally for the image in Figure 7.

Global explanations are shown for the attractiveness classifier in Tables 5, 6. Table 5 shows the effect of singleton interventions in young people, grouped by their gender. The numbers are understood as follows, using the value 0.139 in the first row as an example: This is the probability that a female (Gender = 0) currently classified as unattractive (ŷ = 0) not having 5 o'Clock Shadow ( $[eqn]$ would be classified as attractive (ŷ = 1) if she did ( $[eqn]$ ). The table indicates that females improve their chance of being classified as attractive the most if they remove a possible Big Nose, whereas the best strategy for men is to obtain a Pointy Nose if they do not already have one. For comparison, concept attributions using TCAV scores (Kim et al., 2018) are included in Table 7. These values reflect how concept interventions are moving the prediction toward the subject being seen as more attractive. According to the TCAV scores, obtaining Arched Eyebrows is most beneficial for both genders, followed by the removal of Bushy Eyebrows. The differences in attribution from the two approaches are to be expected. TCAV concept directions are identified with labeled data per concept, with no causal considerations. Sufficiency scores, however, directly reference the probability of crossing a decision boundary, with sound causal interpretation. As an example, the sufficiency attributions for Arched Eyebrows in Table 5 are estimated based on controlled generation of data where only the eyebrows change, thus reflecting the isolated effects of this change. The TCAV scores using observed data (conditioning on a value for Arched Eyebrows) are more prone to influence from correlated concepts. This could be an explanation for the contrasting results obtained for this concept by the two approaches.

Note that the sufficiency scores also better reflect subgroup tendencies, with the effect of interventions to, e.g., Big Nose, Arched Eyebrows and Smiling being more distinct when comparing the male and female subgroups, than what is reflected by the TCAV scores.

Lastly, Table 6 is included as an example of an explanation based on an intervention to a non-leaf variable in the explanation graph for the attractiveness classifier. For young females without makeup, the intervention Makeup = 1 is considered. It is shown that accounting for the causal influence of Makeup on the concepts representing appearance of Pointy Nose and Arched Eyebrows, attributes a higher effect of Makeup on the attractiveness prediction, as measured by the probability of sufficiency, compared to the independent intervention. The probability of sufficiency for Pointy Nose = 1 in particular, shown in Table 5, suggests that Pointy Nose = 1 increases the chance of predicted attractiveness, which is accounted for by the FSCM-based calculation of the probability of sufficiency of Makeup = 1. Independent intervention on Makeup, however, will not influence the value of Pointy Nose, and so the appearance of a pointy nose in the generated image will not change. The PSCM interval included in Table 6 further suggest the independence-model underestimates the true causal effect of Makeup compared to all possible FSCMs compatible with the data, with the independent attribution notably lower than that of the lower PSCM bound.

Discussion

5.3

Assumptions

5.3.1

Section 5.2 presents a set of results obtained by implementing and testing a proof-of-concept explanation model for classifiers trained on the CelebA dataset. As discussed in Section 1, understandability and fidelity of explanations are contingent on the viability of the model assumptions under which explanations are generated. Thus, this section discusses the assumptions made for the proof-of-concept model presented, and their implications regarding explanation interpretation. While the design of this model is intentionally simplistic in order to feasibly illustrate the potential of the general framework, implementation-specific weaknesses are discussed to highlight challenges that may arise when model assumptions are violated.

The explanation vocabulary

5.3.1.1

The CelebA dataset includes labels for 40 binary attributes. To feasibly model the SCM over the explanation vocabulary z, a subset of these 40 attributes is chosen for z, as detailed in Section 5.1. The remaining attributes are considered part of w. Two assumptions are made for this to be meaningful. The first is that the attributes included in z are independent of those left to be encoded in w, such that the effects of interventions in z are reliably mapped to x via α. The second is that the selection of concepts for z is reasonable for explaining h in areas of interest. As this is typically the decision boundary of h, this means that z should include those concepts that, when intervened upon, will move the corresponding input across this boundary (locally or globally). This selection should however not be influenced by expectations regarding which concepts should ideally guide h, as such a restriction for z could prevent the framework from detecting unexpected and potentially unwanted behavior in h.

The causal model

5.3.1.2

Given a selection of variables z now assumed to be independent of all other information present in x, the assumption that the causal structure and the structural equations of the SCM M accurately reflect the world, concludes the vocabulary context. The causal structure in the CelebA examples are mainly learned from data, by detecting (conditional) dependence among pairs of variables and querying an expert (an LLM or the model designer) for the most likely causal direction of this dependence. This results in substructures such as Gender → Smiling, which could be argued not a true causal process. While a person's gender is not expected to have direct causal influence on the likelihood of them smiling, the dependence in the data is more likely a result of either an indirect effect through a mediator, such as a person's smiling habits being a result of the person's previous experiences with their society's gender biased expectations, or a confounder, such as the data selection process. If the former is the case, it is reasonable to keep the simplified causal component as part of the causal structure, omitting lower level details regarding the unobserved mediating factor. This argument is made in favor of including the components Gender → Smiling and Gender → Makeup in the example. If the latter is the case, one might wish to enforce independence in the final causal structure used for explanation, or otherwise adapt the causal model to include the unobserved confounder. In the discussed examples, the variables Young and Gender are modeled as independent despite dependence present in the data, according to such an argument. As the final choice of causal structure determines how explanations should be interpreted, this highlights the importance of including expert knowledge when designing the causal model.

Finally, when designing the structural equations of the model, reliable expert knowledge is again important. The PSCM intervals included illustrate that data and causal structure are not sufficient for exact counterfactual calculation, and knowledge about causal relationships may improve upon this.

StarGAN concept interventions

5.3.1.3

The StarGAN generator implemented to represent α takes an image x (consistent with concepts z) and a concept vector z′ as input, and generates x′ by minimal alterations to original x such that x′ is consistent with z′. This domain translation is learned by optimizing over the ability of a set of concept classifiers to correctly classify generated samples, including a reconstruction loss encouraging α(x′, z) = x. Thus, w = x which violates the framework assumption w⫫z. It is rather assumed that the design of α ensures that when generating x′ = α(x, z′), all information in x pertaining to original concept values z is replaced by introduced z′, without influence from z. This assumption is validated to the extent of inspection of generated samples.

The focus of the implemented generator is efficient and reliable modeling of concept interventions, leaving state-of-the-art generative quality for future work. While the general quality of the generated images, as illustrated in Figures 6, 7, may have unintended influence on the presented results, this is assumed negligible for the purpose of this proof-of-concept example.

The generator is learned from observations, and is therefore not guaranteed to isolate concepts as intended. For instance, if z = {Young}, it is plausible that the generator may learn to include gray hair as a data instance is mapped from young to old, as these attributes are expected to be dependent. However, if z = {Young, Gray Hair}, the generator is incentivized to avoid confusing the two attributes, as classifiers predicting both attributes influence learning. Because it is assumed that the explanation vocabulary is independent of all other information in x, it is therefore assumed that a single StarGAN generator trained with the complete vocabulary vector z as input, will both distinguish these concepts successfully, as well as avoid that any concept retains information expected to be part of w. This is verified by inspection.

Finally, some of the attributes of the CelebA dataset are highly simplistic, binarized versions of complex concepts, an example of this being the attribute Young representing a person's age as either young or old. Data generation is influenced accordingly. While the original dataset contains people of varied age, generated data are mapped to either young or old as learned binary categories. If an instance x representing a young person is input to the generator with a concept vector where Young = 1 is retained, the expectation is that the age of the generated image will remain unaltered. However, the generator will map the input to the binary representation of young, such that a very young person may be aged, while a close to old person will appear younger in the generated counterpart. Note that in the latter case, this can be especially adverse when explaining misclassified instances. Generated explanations for a person who is close to old may wrongly attribute a change in prediction to a change to a given concept, while it is the slight, unintended age adjustment that ensures the decision boundary is crossed. The results presented in Section 5.2 again assume this effect to be negligible for the data instances chosen to be explained. As this assumption is expected to be violated in a more general context, it illustrates the challenge of ensuring that explanations are generated and interpreted in the appropriate context.

Dataset

5.3.1.4

In the examples presented here, both the model h to be explained and the models that generate the explanations α and (parts of) M are learned from the same dataset. While this is not uncommon for XAI, it is briefly discussed as a potential cause of explanation bias.

The ability to explain decisions is important for many reasons, one of which is to detect unintended behavior of h, as can be a result of learning h from spurious correlations present in biased datasets. In this case, learning α and M from the same dataset could hinder detection of this particular behavior. Considering two concepts that are expected to be independent, but are in fact correlated in the data by some unknown confounder in the data generation process. If during vocabulary selection, one of these concepts are included in z while the other is left in w, the generator may learn a joint version of these concepts while referencing this information by name of the concept in z only. This highlights the importance of a sufficiently extensive vocabulary.

When learning the causal model M, it is typically a combination of data and expert knowledge that decides the final model, such that spurious correlations are more easily detected and removed if present among concepts at this level. For instance, there is a correlation between attributes age and gender in the CelebA dataset. This is identified in the presented example as spurious, such that these two concepts are modeled as independent in the explanation model considered. The ability to incorporate this type of expert knowledge is an important advantage of the causal framework. However, whenever data guides the learning, the causal model risks being biased by spurious correlations similarly to h and α.

Choice of abstraction of model

5.3.2

The level of abstraction of a causal model M determines how the total causal influence of one variable on another is divided between direct and indirect causal influence. If the explanation vocabulary is restricted to be z = {Young}, the causal structure considered is

$$[eqn]$$

modeling the total causal effect as direct effect, abstracting away any mediators.

The example presented in Figure 4, considers the structure

such that some of the total effect is mediated through variables gray hair and glasses.

With the data being images of people limited to visual information, it is conceivable that the vocabulary could be extended further such that all causal effect from young to an image x is mediated through some physical feature explicit in z:

$$[eqn]$$

In all three models, the total causal influence of young on the image and prediction is the same, illustrating that it is a reliable measure independent of abstraction. However, if an explanation model ignores the causal structure by intervening on variables as if independent, only the direct effect is captured, which varies from equal to the total effect for the first model, to 0 for the last model. Thus, direct influence is not a consistent measure.

Discussion

6

The framework for causal post-hoc XAI presented in this study constitutes a substantial extension to a model h to be explained, requiring several additional model components and more extensive datasets than those used to train h. While implementation is more demanding than feature-based or non-causal XAI methods, the main argument in favor of this framework is the understandability it facilitates, both in terms of vocabulary and causal interpretation. The concept vocabulary requires concept information such as labels, but allows explanation of the model in an extended context in which changes to the model input, and subsequently output, has meaningful interpretation. The causal modeling additionally requires expert knowledge, but lets the framework admit counterfactual reasoning within the extended context to identify the causes of model outputs, a property identified as especially advantageous for human explainees (Miller, 2019). As discussed in Section 5.3.2, having a causal model over the considered vocabulary allows estimating the total causal effect between variables, argued a more consistent measure than direct causal effect. This illustrates how even though causal modeling is required for generation of explanations, interpretation of the resulting explanations will require less awareness of model details.

Section 5.3.1 discusses the challenges arising when model assumptions are violated, which suggests that model choices should be made carefully. Whenever model assumptions are found to introduce a bias in explanations, either the model must be adapted or, if not feasible, the explainee must be informed about the biased context in which the explanations are valid. This is, however, not meant to indicate that complex models with less assumptions are always favored over simpler ones. For instance, to ensure that explanations are generated in accordance with their expected causal interpretation, the causal model is primarily required to reflect a level of abstraction consistent with that of the explainee's causal understanding of the world (assumed imprecise but not false). With understandability as a main goal, abstracting away certain low-level singular causal mechanisms is preferable, condensing relevant information into a more accessible representation.

Another significant advantage of the causal explanation model discussed in Section 5.3.1 is that the process of causal modeling of concepts, reliant on some contribution of expert knowledge, is more robust regarding detecting and removing spurious correlations than data-driven modeling. Thus, the explanation model can identify spurious relationships learned by the predictor, even in some cases where the same dataset is used to learn the explanation model. With expert knowledge becoming more accessible through LLMs, as demonstrated with the example presented, this has potential particularly for XAI targeting model improvement through detecting unintended model behavior.

The part of the explanation model that is the most challenging to design is typically the structural equations of the FSCM. Canonical PSCM intervals for counterfactual queries are included in the results presented in Section 5.2 to illustrate what is lost when the true causal mechanisms cannot be determined. For some queries, as seen in the results, the intervals are narrow, suggesting a reasonable approximation of the true value may be given by the interval mean or even by approximate queries using a CBN, as exact model choice is of less consequence. However, this is not guaranteed to be the case. In general, when conditioning on observed events of low probability, a counterfactual probability may range from 0 to 1 depending on the model considered. Two different FSCMs that are both consistent with the data may therefore generate opposite explanations of the same model, and even claim these as certain outcomes. Considering that less likely observations can be especially important to explain reliably, for instance when making sure predictions are meaningful even in areas with limited training data, the general framework still requires access to an FSCM to guarantee arbitrary query calculation. Note that non-canonical PSCMs defined by partial expert knowledge may reduce the size of the intervals, extending the scope of queries that may be approximated. Future work may further compare the feasibility and capacity of various approaches to exact and approximate calculations.

While the types of explanations discussed in this work are based on calculating the probability of sufficiency of concept interventions, this is not a limitation of the framework, and future work on extending the types of explanations considered may explore probabilities of necessity, as well as considering explanations exploring actual causation in line with Beckers (2022).

Other lines of future work include developing general architectures for modeling the concept-to-data generative process, respecting the requirements detailed for the concept space while improving generative quality. Automated selection and detection of relevant concepts for the explanation vocabulary may be further explored by inspection of the model to be explained prior to designing the causal model for explanation. In a similar vein, comparing concept attributions can guide the selection of a reasonable vocabulary from a larger set of predefined concepts. Techniques for concept detection such as sparse autoencoders, that have been shown to be useful for detecting interpretable concepts without access to human-annotated data (Cunningham et al., 2023), may further help suggest relevant concepts to be included in the vocabulary without being influenced by preconceptions about concept importance. Automated concept detection could help ensure that the vocabulary contains the most relevant concepts by considering how well they approximate the original model, and whether interventions to the concepts are likely to cross a decision boundary.

Additionally, existing concept attribution methods may be used to guide the search for selecting optimal explanations. Specifically, the number of counterfactuals that can be considered grows exponentially in the number of concepts, for instance 2^|z|^ for binary concepts. In those computationally heavy cases, concept attribution can serve as a heuristic, for example, by ordering concepts to intervene on, combined with early stopping strategies.

Finally, future work on testing the quality of the explanations is necessary. This in particular applies to the choice of causal model, which because it relies on expert knowledge, is required verified for each application. The wrong choice of causal model will, in the worst case, compromise both fidelity and understandability. First, to ensure fidelity, the chosen causal model must align as closely as possible with the ground truth causal mechanisms governing the environment modeled, to the chosen degree of abstraction. The comparison of causal calculations with independent concepts and with a more complex causal structure presented in Section 5.2 provides a preliminary illustration regarding the influence of the choice of causal structure. The PSCM intervals further illustrate that even assuming a correctly designed structure, multiple models can be considered with different structural function configurations, which give inconsistent explanations. The sensitivity of the explanation framework with regards to the choice of structural equations is given by the size of the PSCM interval, and is thus quantifiable. Future work is needed to analyze the sensitivity of the framework with regards to the choice of variables and their structure.

To ensure understandability, the model should align with the user's mental causal model. Ideally, the user's model aligns with the ground truth, presenting a unified goal. If this is not the case, however, fidelity should not be compromised. Further research employing user studies can investigate how to ensure that the explanations are presented for optimal understandability with reference to the causal model of the framework, if this model deviates from the user's mental model.

Conclusion

7

This study introduces a conceptual framework for causal concept-based XAI, that is designed to allow generation of causal and counterfactual explanations in a concept vocabulary. The framework consists of a causal model over the concepts in the vocabulary, and a causal generator that maps concepts to data features. The generation of local and global explanations using this framework, based on the probability of sufficiency of concept interventions, is detailed. A proof-of-concept explanation model is applied to classifiers trained on the CelebA dataset, and a selection of explanations generated is presented. These results show promise and motivate future work toward improved realization of the conceptual framework in generalized domains, and toward testing the achieved understandability and fidelity of the explanations.