Towards a fuller understanding of neurons with Clustered Compositional Explanations

TL;DR

This paper introduces Clustered Compositional Explanations, a method that extends existing explanations of neuron behavior by incorporating clustering and heuristics to cover a wider range of activations, enhancing interpretability.

Contribution

It proposes a novel generalization of Compositional Explanations that captures broader neuron behaviors through clustering and heuristic search, addressing previous limitations.

Findings

Broader spectrum of neuron behaviors approximated

New heuristics improve explanation quality

Framework for evaluating explanation algorithms

Abstract

Compositional Explanations is a method for identifying logical formulas of concepts that approximate the neurons' behavior. However, these explanations are linked to the small spectrum of neuron activations (i.e., the highest ones) used to check the alignment, thus lacking completeness. In this paper, we propose a generalization, called Clustered Compositional Explanations, that combines Compositional Explanations with clustering and a novel search heuristic to approximate a broader spectrum of the neurons' behavior. We define and address the problems connected to the application of these methods to multiple ranges of activations, analyze the insights retrievable by using our algorithm, and propose desiderata qualities that can be used to study the explanations returned by different algorithms.