From Neural Activations to Concepts: A Survey on Explaining Concepts in Neural Networks

TL;DR

This survey reviews recent methods for explaining and integrating concepts within neural networks, highlighting their role in advancing neuro-symbolic AI by linking learning and reasoning.

Contribution

It provides a comprehensive overview of approaches for explaining and inserting concepts in neural networks, emphasizing their importance for neuro-symbolic AI development.

Findings

Concept explanations facilitate reasoning integration.

Knowledge extraction and insertion enhance neural network interpretability.

Survey highlights key methods and future directions in concept-based explainability.

Abstract

In this paper, we review recent approaches for explaining concepts in neural networks. Concepts can act as a natural link between learning and reasoning: once the concepts are identified that a neural learning system uses, one can integrate those concepts with a reasoning system for inference or use a reasoning system to act upon them to improve or enhance the learning system. On the other hand, knowledge can not only be extracted from neural networks but concept knowledge can also be inserted into neural network architectures. Since integrating learning and reasoning is at the core of neuro-symbolic AI, the insights gained from this survey can serve as an important step towards realizing neuro-symbolic AI based on explainable concepts.