Open Vocabulary Compositional Explanations for Neuron Alignment

TL;DR

This paper introduces a flexible framework for generating open vocabulary compositional explanations of neuron activations in vision models, enabling analysis with arbitrary concepts beyond predefined datasets.

Contribution

It presents a novel method leveraging open vocabulary semantic segmentation masks to produce compositional explanations without relying on human-annotated datasets.

Findings

Framework outperforms previous methods in quantitative metrics

Enhances human interpretability of neuron explanations

Allows probing neurons with arbitrary concepts and datasets

Abstract

Neurons are the fundamental building blocks of deep neural networks, and their interconnections allow AI to achieve unprecedented results. Motivated by the goal of understanding how neurons encode information, compositional explanations leverage logical relationships between concepts to express the spatial alignment between neuron activations and human knowledge. However, these explanations rely on human-annotated datasets, restricting their applicability to specific domains and predefined concepts. This paper addresses this limitation by introducing a framework for the vision domain that allows users to probe neurons for arbitrary concepts and datasets. Specifically, the framework leverages masks generated by open vocabulary semantic segmentation to compute open vocabulary compositional explanations. The proposed framework consists of three steps: specifying arbitrary concepts, generating semantic segmentation masks using open vocabulary models, and deriving compositional explanations from these masks. The paper compares the proposed framework with previous methods for computing compositional explanations both in terms of quantitative metrics and human interpretability, analyzes the differences in explanations when shifting from human-annotated data to model-annotated data, and showcases the additional capabilities provided by the framework in terms of flexibility of the explanations with respect to the tasks and properties of interest.