Regional, Lattice and Logical Representations of Neural Networks

TL;DR

This paper introduces an algorithm to translate neural networks with ReLU activations into regional, lattice, and logical representations, enhancing interpretability by analyzing the network's piecewise linear structure.

Contribution

The paper presents a novel algorithm for converting neural networks into interpretable regional representations and empirically studies their complexity and properties.

Findings

Algorithm effectively translates neural networks into regional representations.

Complexity of regional representations varies with network size.

Translations often satisfy properties enabling lattice and logical representations.

Abstract

A possible path to the interpretability of neural networks is to (approximately) represent them in the regional format of piecewise linear functions, where regions of inputs are associated to linear functions computing the network outputs. We present an algorithm for the translation of feedforward neural networks with ReLU activation functions in hidden layers and truncated identity activation functions in the output layer. We also empirically investigate the complexity of regional representations outputted by our method for neural networks with varying sizes. Lattice and logical representations of neural networks are straightforward from regional representations as long as they satisfy a specific property. So we empirically investigate to what extent the translations by our algorithm satisfy such property.