Attributing Learned Concepts in Neural Networks to Training Data

TL;DR

This paper investigates which training data most influence learned concepts in neural networks, using data attribution and probing methods, revealing that concepts are formed from diffuse, robust features rather than specific examples.

Contribution

It introduces a combined approach using data attribution and concept probing to identify influential training data for neural network concepts.

Findings

Removing top attributing images does not alter concept location

Concept features are diffusely spread across training data

Concept formation shows robustness and convergence

Abstract

By now there is substantial evidence that deep learning models learn certain human-interpretable features as part of their internal representations of data. As having the right (or wrong) concepts is critical to trustworthy machine learning systems, it is natural to ask which inputs from the model's original training set were most important for learning a concept at a given layer. To answer this, we combine data attribution methods with methods for probing the concepts learned by a model. Training network and probe ensembles for two concept datasets on a range of network layers, we use the recently developed TRAK method for large-scale data attribution. We find some evidence for convergence, where removing the 10,000 top attributing images for a concept and retraining the model does not change the location of the concept in the network nor the probing sparsity of the concept. This suggests that rather than being highly dependent on a few specific examples, the features that inform the development of a concept are spread in a more diffuse manner across its exemplars, implying robustness in concept formation.