Some Theorems for Feed Forward Neural Networks

TL;DR

This paper introduces 'Orientation Vectors' as a new efficient method for training feed forward neural networks, especially in high-dimensional, sparse clustering problems, reducing computational complexity and offering insights for deep learning architectures.

Contribution

The paper presents a novel 'Orientation Vectors' method for neural network training that is less computationally intensive and scalable, with theoretical proofs and practical demonstrations.

Findings

Method reduces computational effort compared to Radial Basis Function methods.

Network size grows logarithmically with the number of clusters.

Applicable to deep learning architectures with invertibility properties.

Abstract

In this paper we introduce a new method which employs the concept of "Orientation Vectors" to train a feed forward neural network and suitable for problems where large dimensions are involved and the clusters are characteristically sparse. The new method is not NP hard as the problem size increases. We `derive' the method by starting from Kolmogrov's method and then relax some of the stringent conditions. We show for most classification problems three layers are sufficient and the network size depends on the number of clusters. We prove as the number of clusters increase from N to N+dN the number of processing elements in the first layer only increases by d(logN), and are proportional to the number of classes, and the method is not NP hard. Many examples are solved to demonstrate that the method of Orientation Vectors requires much less computational effort than Radial Basis Function methods and other techniques wherein distance computations are required, in fact the present method increases logarithmically with problem size compared to the Radial Basis Function method and the other methods which depend on distance computations e.g statistical methods where probabilistic distances are calculated. A practical method of applying the concept of Occum's razor to choose between two architectures which solve the same classification problem has been described. The ramifications of the above findings on the field of Deep Learning have also been briefly investigated and we have found that it directly leads to the existence of certain types of NN architectures which can be used as a "mapping engine", which has the property of "invertibility", thus improving the prospect of their deployment for solving problems involving Deep Learning and hierarchical classification. The latter possibility has a lot of future scope in the areas of machine learning and cloud computing.