Understanding Inhibition Through Maximally Tense Images

TL;DR

This paper introduces a novel approach to understanding feature inhibition in vision models by analyzing maximally tense images (MTIs) and developing visualization techniques to interpret excitatory and inhibitory features.

Contribution

It proposes the concept of MTIs and new visualization methods to study feature inhibition, addressing limitations of existing interpretability tools in neural networks.

Findings

MTIs reveal how images can simultaneously excite and inhibit features.

New visualization techniques effectively separate excitatory and inhibitory components.

Superposition of features complicates interpretation of inhibition mechanisms.

Abstract

We address the functional role of 'feature inhibition' in vision models; that is, what are the mechanisms by which a neural network ensures images do not express a given feature? We observe that standard interpretability tools in the literature are not immediately suited to the inhibitory case, given the asymmetry introduced by the ReLU activation function. Given this, we propose inhibition be understood through a study of 'maximally tense images' (MTIs), i.e. those images that excite and inhibit a given feature simultaneously. We show how MTIs can be studied with two novel visualization techniques; +/- attribution inversions, which split single images into excitatory and inhibitory components, and the attribution atlas, which provides a global visualization of the various ways images can excite/inhibit a feature. Finally, we explore the difficulties introduced by superposition, as such interfering features induce the same attribution motif as MTIs.