Single neuron-based neural networks are as efficient as dense deep neural networks in binary and multi-class recognition problems

TL;DR

This study shows that sparse, single neuron-based neural networks can match the performance of dense networks in recognition tasks, challenging the assumption that complexity is necessary for high accuracy.

Contribution

It demonstrates that single neuron networks are capable of high-dimensional recognition, offering a simpler alternative to dense neural networks.

Findings

Sparse networks perform as well as dense networks in recognition tasks.

Single neuron networks excel in binary classification.

Combined single neuron networks achieve competitive multi-class recognition results.

Abstract

Recent advances in neuroscience have revealed many principles about neural processing. In particular, many biological systems were found to reconfigure/recruit single neurons to generate multiple kinds of decisions. Such findings have the potential to advance our understanding of the design and optimization process of artificial neural networks. Previous work demonstrated that dense neural networks are needed to shape complex decision surfaces required for AI-level recognition tasks. We investigate the ability to model high dimensional recognition problems using single or several neurons networks that are relatively easier to train. By employing three datasets, we test the use of a population of single neuron networks in performing multi-class recognition tasks. Surprisingly, we find that sparse networks can be as efficient as dense networks in both binary and multi-class tasks. Moreover, single neuron networks demonstrate superior performance in binary classification scheme and competing results when combined for multi-class recognition.

Tables

Table 1. TABLE I : Performance comparison between single, 10, and 100 neurons networks over random generated data.

Network Size		Single Neuron Net			10-Neurons Net			100-Neurons Net
Sample Size		${10}^3$	${10}^4$	${10}^5$	${10}^3$	${10}^4$	${10}^5$	${10}^3$	${10}^4$	${10}^5$
	Accuracy	0.693	0.727	0.703	0.771	0.784	0.782	0.753	0.778	0.782
	Sensitivity	0.695	0.727	0.704	0.775	0.783	0.73	0.758	0.779	0.781
$𝒩(0,0.1)$	Specificity	0.691	0.728	0.702	0.768	0.786	0.782	0.750	0.777	0.782
	Accuracy	0.706	0.698	0.700	0.772	0.781	0.782	0.760	0.778	0.782
	Sensitivity	0.694	0.698	0.701	0.760	0.782	0.783	0.756	0.778	0.781
$𝒩(0,0.5)$	Specificity	0.720	0.698	0.700	0.786	0.782	0.780	0.765	0.778	0.782
	Accuracy	0.701	0.6936	0.700	0.776	0.780	0.781	0.756	0.779	0.781
	Sensitivity	0.699	0.693	0.700	0.775	0.785	0.781	0.767	0.777	0.781
$𝒩(0,1.0)$	Specificity	0.703	0.693	0.699	0.778	0.775	0.781	0.747	0.780	0.782

Table 2. TABLE II : Performance of tearing down multi-class problems into binary classification with the use os small sized hidden layers.

Dataset	MNIST/Single Neuron Net		CIFAR-10/128 Neurons Net
	Zero against all	0.982	Airplane against all	0.959
	One against all	0.985	Automobile against all	0.976
	Two against all	0.978	Bird against all	0.952
	Three against all	0.960	Cat against all	0.948
	Four against all	0.997	Deer against all	0.955
	Five against all	0.985	Dog against all	0.944
	Six against all	0.985	Frog against all	0.960
	Seven against all	0.975	Horse against all	0.963
	Eight against all	0.956	Ship against all	0.977
Accuracy	Nine against all	0.932	Truck against all	0.961

Full text

11footnotetext: Research reported in this publication was supported by the National Science Foundation under the CAREER Award Number 1652203. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Science Foundation.22footnotetext: Yassin Khalifa, Justin Hawks and Ervin Sejdić are with the Department of Electrical and Computer Engineering, Swanson School of Engineering, University of Pittsburgh, Pittsburgh, PA, 15261, USA (e-mail: esejdic@ieee.org).

I Introduction

Over the last several years, machine learning techniques, particularly neuromorphic computing architectures, have been used to mimic the challenging pattern recognition abilities of biological systems [1]. In fact, it could be argued that neural networks based systems are learning feature hierarchies automatically so that they are replacing many learning models that depend on hand-designed heuristics [1, 2, 3]. This is achieved through multiple levels of representations generated by successive non-linear modules, each of which transforms low level representations into more abstract high level representations [1, 4, 5, 6, 7]. Theoretical results have shown that deep architectures may be needed to efficiently model complex functions and represent high levels of abstraction required for challenging recognition and AI tasks [8, 9, 5, 10].

Many questions come up when choosing an architecture to address a practical problem, including how many layers we should use and how large each layer is supposed to be [11, 12]. Network capacity increases with both number and size of hidden layers due to the fact that neurons collaborate to express complex functions and achieve better generalization [11]. However, high capacity models may fit also dataset outliers and lead to overfitting. Although, one can always use shallow networks to avoid overfitting, but sometimes data complexity necessitates using deep architectures [11, 13]. Multiple solutions have been suggested though to avoid overfitting in deep networks including dropout [14], regularization, and noisy inputs [11, 15, 13].

In practice, overfitting control is preferred over network size tuning as it was found that smaller networks are hard to train using local optimizers like gradient descent [13, 16, 17, 18]. Obviously, smaller networks will have fewer local solutions with easy convergence, however, most of these solutions are unreliable (high loss) especially when using batch optimization [13]. This also makes small sized networks vulnerable to huge loss variance influenced by random initializations [11]. It is hard to prove this mathematically due to the poor understanding of loss function and its shape for neural networks [13]. On the other hand, simplifying the problem through conquering into smaller problems or multiple recognition stages may help build efficient, yet cheap to train models.

Understanding the response of individual neurons in parts of brain that are highly believed to be responsible for visual object recognition, is quite challenging and hard to predict [19, 20, 21]. Nevertheless, it is known that they are activated by a set of complex visual features and, thus they cannot be narrowly tuned as detectors for specific objects [19, 22, 23]. Therefore, single neurons may not be acting as sparse object detectors, but, rather as elements of a group that as a whole, provides object recognition [19, 22]. However, they are often able to maintain preferences among objects like changes in size and shape which is called neural tolerance [19, 20]. This suggests that single neurons together decode the object space and identity including position, size, context, and other variables to achieve powerful representation and avoid binding this information at successive stages [19, 24, 25, 26]. As an analogy of how brain single neurons behave in visual recognition tasks, artificial neural networks can make use of it as a design concept in developing populations of single neuron or small-sized networks for object recognition.

The design concept introduced here, has proven effectiveness for many classifiers and configurations including SVM and neural networks [27, 28, 29, 30]. The reason it is often successful for classification lies in the fact that it is easier to build classifiers to separate two classes rather than for multiple classes [28]. Usually, the approach is referred to as decomposition [28], binarization [28], or modularization [30] and has been addressed in different ways. The first used decomposition strategy is called ”one-vs-one” (OVO) and it divides the problem into as many binary classification problems as the number of unique combinations between each two classes [28, 29]. The second strategy is called ”one-vs-all” (OVA) and it trains a classifier to discriminate each class from all other classes [28, 29].

Modularization in neural networks has been introduced by [29] through attempting to separate each class from all other classes and then subsequently pairwise separate each class using subnetworks. Furthermore, [30] used K networks to reduce a K-class problem into a set of K two class problems, however they focused more on the back-propagation algorithm and enhancing its convergence speed. In this work, we used an OVA approach as in [30] and we focused more about the size of modular networks used for each binary sub-classification problem and the performance of the network.

This paper addresses the feasibility of neuron population based design approach through answering three main questions: 1. How does the model performance change with the size of hidden layer in the used network? 2. Is it possible that single neuron or small-sized networks can achieve good performance with the binary sub-classification problems of high dimensional multi-class data? 3. Can binary classification networks with low number of hidden neurons (especially single neuron networks) be utilized to perform multi-class recognition? and what is the efficiency of these systems compared to multi-class dense systems? To answer these questions, we experimented multiple architectures along with different levels of abstraction in recognition tasks for three datasets.

II Methods

II-A Datasets

To conduct experimental results about the main objectives of this work, we used three main datasets for all trials. The first dataset is composed of 3 dimensional data points drawn from a normal distribution with different values of mean and standard deviation. Particularly, we used a combination of a single value for mean (0) and 3 values for standard deviation (0.1, 0.5, 1). Multiple numbers of data points were used as well for each set, which gives a total of nine sets (3 sets of different lengths drawn from each distribution). Two categories were created inside the generated sets through positively and negatively biasing samples of the data with the same amount. Fig. 1(a-d) shows a sample of the generated sets with a mean of 0 and standard deviation of 1. The categories for this set were created by adding $0.5$ to half of the set and subtracting the same value from the other half.

The second dataset is the famous MNIST [2], widely used for handwritten digit recognition benchmarking. MNIST consists of two sets of images, the first set is 60,000 examples for training and a test set of 10,000 examples. All digits are size-normalized and centered in $28\times 28$ gray scale images (Fig. 1(e)). MNIST dataset doesn’t require formatting or preprocessing which makes it optimal for testing learning techniques and new recognition algorithms [2]. The third dataset is the CIFAR-10 dataset [31] which consists of 60,000 color images each of $32\times 32$ pixels in $10$ classes (Fig. 1(f)). The dataset is divided into two sets, one for training (50,000 images) and another for testing (10,000 images).

II-B CNN Design

The main architecture of CNN used for all multi-class recognition experiments performed in this analysis, included two convolutional layers, two max pooling layers (one after each convolutional layer), a single hidden dense layer (ReLU activated), and an output layer with a number of units equal to the number of classes in the target dataset. The first convolutional layer applies 32 $5\times 5$ filters with ReLU activation followed by a $2\times 2$ filter max pooling with strides of 2 (non overlapping pooled regions). The second convolutional layer applies 64 $5\times 5$ filters with ReLU activation followed by a $2\times 2$ filter max pooling with strides of 2. For the hidden layer, we used $1,2,4,8,16,\dots,1024$ units for the different experiments performed. All experiments developed for this analysis were implemented using TensorflowTM and Matlab; and tested over an NVIDIA Tesla M40 GPU.

II-C Effect of Layer Size

The number of hidden neurons used in different network architectures is assessed here through running multiple trials over the 3 datasets mentioned before. First, we trained three feedforward networks for a simple binary classification task over all of the randomly generated sets. Each set was divided into 80% for training and 20% for testing of its length. The three networks are of one hidden layer but one with a single neuron, the second with 10 neurons, and the last one with 100 neurons. The average of results was taken over multiple iterations (100 to 1000) to avoid randomness of the experiment [32].

Second, to see the effect of the number of hidden neurons on more complex recognition problems, we trained a convolutional neural network (CNN) for MNIST handwritten digit recognition and another for CIFAR-10 content recognition. Both networks have two convolutional layers, two pooling layers (one after each convolutional layer), and a single hidden dense layer (ReLU activated). The first convolutional layer applies 32 $5\times 5$ filters with ReLU activation followed by a $2\times 2$ filter max pooling with strides of 2 (non overlapping pooled regions). The second convolutional layer applies 64 $5\times 5$ filters with ReLU activation followed by a $2\times 2$ filter max pooling with strides of 2. For the hidden layer, we used $1,2,4,8,16,\dots,1024$ in order to examine the effect of layer size on the recognition task in both datasets.

II-D Effect of Tearing down the Recognition Problem

As mentioned before, dense and deep networks may be needed to model complex recognition problems. However, this will come with a cost, overfitting and expensive training. So, here we test the ability of tearing down multi-class recognition problems into a series of binary classifiers. MNIST and CIFAR-10 datasets were used also for this experiment but with converting each of them into 10 different datasets from the labels point of view. In other words, the labels of each dataset were altered 10 times to classify only one category out of the 10 categories against all the others. For MNIST dataset we have 10 classification problems, the first is to classify the [math] digit against $1,\ 2,\ \dots,\ and\ 9$, the second is to classify the $1$ digit against $0,\ 2,\ \dots,\ and\ 9$, and so on. The same also is performed for the CIFAR-10 dataset but considering its different categories. All generated datasets were balanced in terms of classes before being used in training.

We used the same convlutional network mentioned before and tried different sizes for hidden layer too. Ten networks were trained for each dataset and the results were compared with small sized, full classification networks tested with the same datasets. In order to use these parallel networks for 10 classes recognition, the input image is administered to each of the 10 networks and the resultant digit is represented by the network that gives a positive response as in Fig. 2. Due to the independence of the 10 networks, there might be redundancy in the output (i.e. an image identified more than once throughout the 10 networks). This can be solved through priority encoding the results (take the first positive result and ignore the rest of networks) or consider this as misclassification. For this study, we considered redundancy as misclassification. We saw it as a more convenient way for performance comparison between different architectures.

III Results

Changing the hidden layer size in a neural network will definitely affect the performance of the network, but, the question is whether the change in performance is worth it or not. Testing networks of different hidden layer sizes using the random generated data gave the results shown in Table I. On the other hand, testing over a wider range of hidden layer sizes for multi-class complex data showed that increasing the number of hidden neurons is not quite effective after a certain point. The accuracy of 10 classes recognition task for both MNIST and CIFAR-10 remains nearly constant after a hidden layer size of 128 neurons as shown in Fig. 1(g). Probably this size will be different from a dataset to another and even from a recognition task to another, however, this shows that the same performance can be achieved with way smaller networks. No changes in the total loss pattern were observed as well after this layer size as shown in Fig. 1(h) and (i). In addition to that, small sized networks were found to suffer from high loss despite of faster convergence which can be clearly seen in CIFAR-10 total loss in Fig. 1(i).

Since over-reduction of network size alone, shows poor performance in multi-class recognition problems as it appears in Fig. 1(g), simplifying the recognition problem might be the solution. Given the performance of single neuron networks for binary classification in Table I, we test the ability of tearing down multi-class recognition problems into a set of binary classifiers using small sized networks as in Fig. 2. Table II shows a sample of the results for tearing down the MNIST and CIFAR-10 datasets into 10 different binary classification problems. It seems that using a small sized hidden layer along with a binary problem gives better results than the best performance achieved using high density layers used with multi-class recognition problems considering all classes. For MNIST problem, good results were achieved using a single neuron in hidden layer. On the other hand, CIFAR-10 classification accuracy did not jump over 80% before using 64 neurons in the hidden layer.

The 10 networks trained for binary classification problems for MNIST were combined together to form a 10-classes recognizer as in Fig. 2. The new system correctly identified 84.21% of the test data, produced multiple classification results for 12.79% of the test data (maximum of two positive results per image appeared), and the rest of test data (3%) were misclassified. For CIFAR-10 dataset, to achieve the same good results as in MNIST experiment, we used 10 networks with at least 128 neurons per hidden layer for each, which will be more complex and ineffective than using a full classification network with the same number of neurons in hidden layer.

IV Discussion

The experimental results of this work, showed that neural networks of dense hidden layers might not be of a great help to achieve the desired modeling of the recognition/classification task. For a binary classification task, increasing the hidden layer size did not add much to all the aspects of system performance as shown in Table I. Even in complex multi-class recognition tasks like digits and objects identification, the performance becomes nearly the same after a certain layer size. This characterizing layer size will probably depend on the level of abstraction of the assessed problem, however, we can clearly see that good performance can be achieved by using fairly sparse networks.

An acceptable performance can be easily achieved in simple classification tasks using small sized networks and becomes harder in high dimensional tasks. This can be noticed through the differences between training networks for binary classification task and multi-class tasks (Table I and Fig. 1(g)). The classification accuracy is nearly stable at low network density for the binary random data, but needs more hidden neurons to get to the same stable performance for MNIST and CIFAR-10 tasks. This suggested tearing down multi-class recognition tasks to multiple binary classification tasks for which, fast convergence, more simple architectures, and acceptable performance can be reached easier.

Using the binary classification scheme for both MNIST and CIFAR-10 datasets, gave superior performance even with using a single neuron hidden layer. Compared to the highest accuracy achieved for 10-classes recognition, the binary scheme achieved higher classification accuracy for all components. This proves our claim about using populations of binary systems to represent higher dimensional datasets for a better performance and cheap training.

The high accuracy achieved in binary classification networks pushed toward building multi-class recognition based on these networks. Parallel sparse hidden layer, binary networks with the same number as desired classes, were used to build multi-class classifiers with a higher accuracy than a single multi-class network with the same number of hidden neurons used for the same task. A single network with 16 neurons in the hidden layer got a classification accuracy of 82% (Fig. 1(g)) for MNIST dataset while 10 single neuron binary networks achieved around 84% accuracy. The low value of contradicting results from each of the 10 combined networks comes from the fact that the accuracy of each single network is very high that there is a very low chance that an image will get identified in more than one network.

The experiments performed in this study, showed that robust results can be achieved using small number of hidden neurons and these results were confirmed by multiple trials on different datasets. The experiments showed also that using neuron populations in artificial object recognition can achieve a similar performance pattern as anticipated from the biological models. However, it must be taken in consideration that we tested the design concept for multi-object recognition and not for the same object attributes like shape, size, color, and rotation. This can be an indicator that increasing the level of representation may be in favor of the recognition performance. In other words, adding a neuron to the population to tolerate more visual changes of objects may help achieve better recognition performance between objects.

To conclude, we assessed the effect of reducing the hidden layer size in neural networks on the performance of different recognition tasks including binary random data and 10-classes recognition problems. The results showed that high performance can be achieved using fairly small-sized networks from the number of hidden neurons prospective. Moreover, we assessed the use of a population of small sized binary networks in building multi-class recognition systems and it showed superior results compared to multi-class systems with the same hidden layer size. There is more to build over these preliminary findings, therefor, we intend to test more visual aspects in object recognition with neuron populations for better generalization of the proposed design concept.