Linear Independence of Generalized Neurons and Related Functions

TL;DR

This paper characterizes when neurons with arbitrary layers and widths are linearly independent as parameters vary, focusing on generic analytic activation functions, extending previous results beyond two-layer networks.

Contribution

It provides a complete characterization of linear independence for neurons with arbitrary architecture and generic analytic activations, generalizing prior two-layer results.

Findings

Complete characterization for generic analytic activation functions.

Extension of linear independence results to deep and wide networks.

Simplified criteria for linear independence in neural networks.

Abstract

The linear independence of neurons plays a significant role in theoretical analysis of neural networks. Specifically, given neurons $H_1, ..., H_n: \bR^N \times \bR^d \to \bR$, we are interested in the following question: when are $\{H_1(\theta_1, \cdot), ..., H_n(\theta_n, \cdot)\}$ are linearly independent as the parameters $\theta_1, ..., \theta_n$ of these functions vary over $\bR^N$. Previous works give a complete characterization of two-layer neurons without bias, for generic smooth activation functions. In this paper, we study the problem for neurons with arbitrary layers and widths, giving a simple but complete characterization for generic analytic activation functions.