The inverse problem for neural networks

TL;DR

This paper investigates the inverse problem for neural networks with piecewise-affine activations, focusing on computing preimages of sets to enhance analysis and interpretability of neural models.

Contribution

It revisits a classical result on preimages of polyhedral sets and demonstrates their effective computation for neural network analysis.

Findings

Preimages of polyhedral sets under neural networks are unions of polyhedral sets.

Effective methods for computing these preimages are presented.

Applications improve neural network interpretability and analysis.

Abstract

We study the problem of computing the preimage of a set under a neural network with piecewise-affine activation functions. We recall an old result that the preimage of a polyhedral set is again a union of polyhedral sets and can be effectively computed. We show several applications of computing the preimage for analysis and interpretability of neural networks.