CRAFT: Concept Recursive Activation FacTorization for Explainability

TL;DR

CRAFT introduces a novel method for explainability that identifies both "what" and "where" in images by generating concept-based explanations, improving faithfulness and human interpretability over existing attribution methods.

Contribution

It proposes a recursive concept detection strategy, a Sobol indices-based importance estimation, and implicit differentiation for concept attribution, advancing automatic concept extraction techniques.

Findings

More faithful concept importance estimation than previous methods

Significant improvement in human-centered utility benchmarks

Effective detection and decomposition of concepts across layers

Abstract

Attribution methods, which employ heatmaps to identify the most influential regions of an image that impact model decisions, have gained widespread popularity as a type of explainability method. However, recent research has exposed the limited practical value of these methods, attributed in part to their narrow focus on the most prominent regions of an image -- revealing "where" the model looks, but failing to elucidate "what" the model sees in those areas. In this work, we try to fill in this gap with CRAFT -- a novel approach to identify both "what" and "where" by generating concept-based explanations. We introduce 3 new ingredients to the automatic concept extraction literature: (i) a recursive strategy to detect and decompose concepts across layers, (ii) a novel method for a more faithful estimation of concept importance using Sobol indices, and (iii) the use of implicit differentiation to unlock Concept Attribution Maps. We conduct both human and computer vision experiments to demonstrate the benefits of the proposed approach. We show that the proposed concept importance estimation technique is more faithful to the model than previous methods. When evaluating the usefulness of the method for human experimenters on a human-centered utility benchmark, we find that our approach significantly improves on two of the three test scenarios. Our code is freely available at github.com/deel-ai/Craft.