FeatInv: Spatially resolved mapping from feature space to input space using conditional diffusion models

TL;DR

This paper introduces FeatInv, a probabilistic method using conditional diffusion models to map neural network features to input images, enhancing interpretability of deep models across various architectures.

Contribution

The paper presents a novel application of conditional diffusion models for high-fidelity, probabilistic feature-to-input mapping in neural networks, improving interpretability.

Findings

Demonstrates excellent reconstruction across CNNs and ViTs.

Validates robustness and qualitative effectiveness of the approach.

Enables visualization of concept steering and feature space analysis.

Abstract

Internal representations are crucial for understanding deep neural networks, such as their properties and reasoning patterns, but remain difficult to interpret. While mapping from feature space to input space aids in interpreting the former, existing approaches often rely on crude approximations. We propose using a conditional diffusion model - a pretrained high-fidelity diffusion model conditioned on spatially resolved feature maps - to learn such a mapping in a probabilistic manner. We demonstrate the feasibility of this approach across various pretrained image classifiers from CNNs to ViTs, showing excellent reconstruction capabilities. Through qualitative comparisons and robustness analysis, we validate our method and showcase possible applications, such as the visualization of concept steering in input space or investigations of the composite nature of the feature space. This approach has broad potential for improving feature space understanding in computer vision models.