Logic interpretations of ANN partition cells

TL;DR

This paper introduces a novel method for interpreting neural networks by translating their partition cells into logical expressions, enabling better understanding of how inputs influence classifications.

Contribution

It develops a logical framework to analyze and manipulate ANN semantics by decomposing input space into partition cells and representing their linear maps as logic expressions.

Findings

Logic expressions effectively represent interaction patterns.

Binary logic trees facilitate interpretation of network behavior.

Method bridges neural network analysis with formal logic tools.

Abstract

Consider a binary classification problem solved using a feed-forward artificial neural network (ANN). Let the ANN be composed of a ReLU layer and several linear layers (convolution, sum-pooling, or fully connected). We assume the network was trained with high accuracy. Despite numerous suggested approaches, interpreting an artificial neural network remains challenging for humans. For a new method of interpretation, we construct a bridge between a simple ANN and logic. As a result, we can analyze and manipulate the semantics of an ANN using the powerful tool set of logic. To achieve this, we decompose the input space of the ANN into several network partition cells. Each network partition cell represents a linear combination that maps input values to a classifying output value. For interpreting the linear map of a partition cell using logic expressions, we suggest minterm values as the input of a simple ANN. We derive logic expressions representing interaction patterns for separating objects classified as 1 from those classified as 0. To facilitate an interpretation of logic expressions, we present them as binary logic trees.