On the space of coefficients of a Feed Forward Neural Network

TL;DR

This paper characterizes the set of all neural networks with different parameters that produce the same function, showing it forms a semialgebraic set, thus providing a geometric understanding of neural network equivalence.

Contribution

It introduces the concept of equivalent neural networks and proves that their coefficient space is a semialgebraic set using the Tarski-Seidenberg theorem.

Findings

The coefficient space of equivalent networks is semialgebraic.

Conditions for network equivalence are established.

The approach uses algebraic geometry to analyze neural network representations.

Abstract

We define and establish the conditions for `equivalent neural networks' - neural networks with different weights, biases, and threshold functions that result in the same associated function. We prove that given a neural network $\mathcal{N}$ with piece-wise linear activation, the space of coefficients describing all equivalent neural networks is given by a semialgebraic set. This result is obtained by studying different representations of a given piece-wise linear function using the Tarski-Seidenberg theorem.