Understanding Task Representations in Neural Networks via Bayesian Ablation

TL;DR

This paper introduces a Bayesian-inspired probabilistic framework to interpret neural network representations, providing tools to analyze their causal roles and properties like distributedness and complexity.

Contribution

It presents a novel probabilistic approach for understanding latent task representations in neural networks, addressing interpretability challenges.

Findings

Provides a suite of metrics for analyzing neural representations.

Defines a distribution over units to infer their causal contributions.

Illuminates properties like distributedness and manifold complexity.

Abstract

Neural networks are powerful tools for cognitive modeling due to their flexibility and emergent properties. However, interpreting their learned representations remains challenging due to their sub-symbolic semantics. In this work, we introduce a novel probabilistic framework for interpreting latent task representations in neural networks. Inspired by Bayesian inference, our approach defines a distribution over representational units to infer their causal contributions to task performance. Using ideas from information theory, we propose a suite of tools and metrics to illuminate key model properties, including representational distributedness, manifold complexity, and polysemanticity.