OMENN: One Matrix to Explain Neural Networks

TL;DR

OMENN introduces a novel method that simplifies neural network explanations by representing each input's processing as a single interpretable matrix, improving transparency and precision in model interpretability.

Contribution

The paper proposes OMENN, a new post-hoc explanation technique that constructs a single matrix to interpret neural network decisions, applicable to modern models like ViTs and CNNs.

Findings

OMENN provides locally precise explanations.

OMENN is competitive with state-of-the-art XAI methods.

Theoretical analysis supports OMENN’s interpretability.

Abstract

Deep Learning (DL) models are often black boxes, making their decision-making processes difficult to interpret. This lack of transparency has driven advancements in eXplainable Artificial Intelligence (XAI), a field dedicated to clarifying the reasoning behind DL model predictions. Among these, attribution-based methods such as LRP and GradCAM are widely used, though they rely on approximations that can be imprecise. To address these limitations, we introduce One Matrix to Explain Neural Networks (OMENN), a novel post-hoc method that represents a neural network as a single, interpretable matrix for each specific input. This matrix is constructed through a series of linear transformations that represent the processing of the input by each successive layer in the neural network. As a result, OMENN provides locally precise, attribution-based explanations of the input across various modern models, including ViTs and CNNs. We present a theoretical analysis of OMENN based on dynamic linearity property and validate its effectiveness with extensive tests on two XAI benchmarks, demonstrating that OMENN is competitive with state-of-the-art methods.