Hidden Activations Are Not Enough: A General Approach to Neural Network Predictions

TL;DR

This paper presents a new mathematical framework using quiver representation theory to analyze neural network predictions, capturing more information than traditional methods and applicable across architectures and tasks.

Contribution

Introduces a novel, architecture- and task-agnostic framework based on quiver representations to analyze neural network predictions and detect adversarial examples.

Findings

Effective adversarial example detection on MNIST and FashionMNIST

Framework captures richer information than hidden activations

Applicable to various architectures and attack methods

Abstract

We introduce a novel mathematical framework for analyzing neural networks using tools from quiver representation theory. This framework enables us to quantify the similarity between a new data sample and the training data, as perceived by the neural network. By leveraging the induced quiver representation of a data sample, we capture more information than traditional hidden layer outputs. This quiver representation abstracts away the complexity of the computations of the forward pass into a single matrix, allowing us to employ simple geometric and statistical arguments in a matrix space to study neural network predictions. Our mathematical results are architecture-agnostic and task-agnostic, making them broadly applicable. As proof of concept experiments, we apply our results for the MNIST and FashionMNIST datasets on the problem of detecting adversarial examples on different MLP architectures and several adversarial attack methods. Our experiments can be reproduced with our \href{https://github.com/MarcoArmenta/Hidden-Activations-are-not-Enough}{publicly available repository}.