Transfer Learning with Sparse Associative Memories

TL;DR

This paper presents a new associative memory-based layer for neural networks that enables incremental learning and real-time training on embedded devices, with slight accuracy trade-offs demonstrated on ImageNet and other datasets.

Contribution

Introduces a novel associative memory layer for neural networks supporting incremental learning and real-time training, enhancing flexibility and speed.

Findings

Supports incremental learning in neural networks

Enables real-time training on embedded devices

Results show slight decrease in accuracy compared to traditional models

Abstract

In this paper, we introduce a novel layer designed to be used as the output of pre-trained neural networks in the context of classification. Based on Associative Memories, this layer can help design Deep Neural Networks which support incremental learning and that can be (partially) trained in real time on embedded devices. Experiments on the ImageNet dataset and other different domain specific datasets show that it is possible to design more flexible and faster-to-train Neural Networks at the cost of a slight decrease in accuracy.

Tables

Table 1. Table 1: Comparison of the accuracy of 128x100-AL with different transfer learning methods on classification task on different domain specific datasets when learning the complete dataset at once. Values are averaged over 30 runs.

		Non incremental methods				Incremental methods
		DNN classifier (SGD)	DNN classifier (SGD + CycleLR)	DNN classifier (Adam)	SVM	K-NN	Proposed model 128x100-AL
Flower102	Top-1	75.71 % ($\pm$ 0.97)	70.52 % ($\pm$ 2.08)	72.12 % ($\pm$ 0.96)	73.80 % ($\pm$ 0.0)	57.13 % ($\pm$ 0.0)	66.92 % ($\pm$ 0.0)
	Top-5	91.89 % ($\pm$ 0.56)	89.07 % ($\pm$ 1.25)	90.17 % ($\pm$ 0.69)	90.94 % ($\pm$ 0.0)	80.34 % ($\pm$ 0.0)	87.48 % ($\pm$ 0.0)
Indoor67	Top-1	65.43 % ($\pm$ 0.83)	64.05 % ($\pm$ 0.93)	61.86 % ($\pm$ 1.48)	67.24 % ($\pm$ 0.0)	52.76 % ($\pm$ 0.0)	56.49 % ($\pm$ 0.0)
	Top-5	90.21 % ($\pm$ 0.50)	89.57 % ($\pm$ 0.56)	87.86 % ($\pm$ 0.52)	90.90 % ($\pm$ 0.0)	78.36 % ($\pm$ 0.0)	87.31 % ($\pm$ 0.0)
CUB200	Top-1	37.31 % ($\pm$ 0.97)	32.69 % ($\pm$ 1.21)	31.97 % ($\pm$ 1.20)	38.02 % ($\pm$ 0.0)	24.70 % ($\pm$ 0.0)	41.51 % ($\pm$ 0.0)
	Top-5	66.74 % ($\pm$ 1.05)	61.83 % ($\pm$ 1.46)	61.61 % ($\pm$ 1.50)	66.74 % ($\pm$ 0.0)	50.31 % ($\pm$ 0.0)	76.18 % ($\pm$ 0.0)
Dog120	Top-1	74.56 % ($\pm$ 0.34)	73.81 % ($\pm$ 0.29)	70.80 % ($\pm$ 0.65)	76.35 % ($\pm$ 0.0)	68.83 % ($\pm$ 0.0)	71.91 % ($\pm$ 0.0)
	Top-5	95.06 % ($\pm$ 0.14)	94.63 % ($\pm$ 0.15)	93.37 % ($\pm$ 0.30)	95.90 % ($\pm$ 0.0)	87.74 % ($\pm$ 0.0)	95.34 % ($\pm$ 0.0)
Stanford40	Top-1	68.05 % ($\pm$ 0.54)	67.81 % ($\pm$ 0.42)	66.21 % ($\pm$ 0.70)	69.54 % ($\pm$ 0.0)	57.36 % ($\pm$ 0.0)	59.06 % ($\pm$ 0.0)
	Top-5	90.67 % ($\pm$ 0.34)	90.61 % ($\pm$ 0.28)	89.46 % ($\pm$ 0.39)	91.68 % ($\pm$ 0.0)	79.10 % ($\pm$ 0.0)	87.02 % ($\pm$ 0.0)

Table 2. Table 2: Comparison of the training time of 128x100-AL with different classifiers on different domain specific datasets when learning the complete dataset at once. Training time duration is expressed relatively to the one of the proposed model: the lower is the value, the better it is. Values are averaged over 30 runs.

	DNN classifier (SGD)	DNN classifier (SGD + CycleLR)	DNN classifier (Adam)	SVM	Proposed model 128x100-AL
Flower102	45 ($\pm$ 8.5)	18 ($\pm$ 3.8)	32 ($\pm$ 5.9)	87 ($\pm$ 5.3)	1
Indoor67	20 ($\pm$ 3.1)	9 ($\pm$ 1.6)	18 ($\pm$ 3.2)	127 ($\pm$ 8.2)	1
CUB200	24 ($\pm$ 3.6)	9 ($\pm$ 1.15)	16 ($\pm$ 2.2)	87 ($\pm$ 3.1)	1
Dog120	18 ($\pm$ 4.1)	9 ($\pm$ 0.5)	15 ($\pm$ 2.2)	214 ($\pm$ 12.4)	1
Stanford40	12 ($\pm$ 2.2)	8 ($\pm$ 0.63)	14 ($\pm$ 2.4)	79 ($\pm$ 5.4)	1

Table 3. Table 3: Comparison of the training time of 128x100-AL with different classifiers on different domain specific datasets when learning the last class of the dataset after having learned the previous classes. Training time duration is expressed relatively to the one of the proposed model: the lower is the value, the better it is. Values are averaged over 30 runs.

	DNN classifier (SGD)	DNN classifier (SGD + CycleLR)	DNN classifier (Adam)	SVM	Proposed model 128x100-AL
Flower102	5410 ($\pm$ 977)	2222 ($\pm$ 428)	3867 ($\pm$ 655)	10460 ($\pm$ 247)	1
Indoor67	1602 ($\pm$ 248)	762 ($\pm$ 120)	1474 ($\pm$ 238)	10207 ($\pm$ 101)	1
CUB200	4632 ($\pm$ 836)	1836 ($\pm$ 271)	3197 ($\pm$ 411)	17111 ($\pm$ 1028)	1
Dog120	2703 ($\pm$ 582)	1333 ($\pm$ 22)	2222 ($\pm$ 337)	31239 ($\pm$ 376)	1
Stanford40	667 ($\pm$ 111)	409 ($\pm$ 16)	737 ($\pm$ 126)	4220 ($\pm$ 44)	1

Equations

$$\mathbf{q}(\mathbf{x})=h(W\cdot \mathbf{x}),$$

$$\forall i,{\mathbf{w}}_i\leftarrow {\mathbf{w}}_i+(\mathbf{x}-{\mathbf{w}}_i)A(t)\Theta (t,i^{\ast },i).$$

$$\Omega =\underset{(x,y)\in \mathcal{X}}{\max }{\mathbf{e}}_y\cdot Q(\mathbf{x}{)}^{\top }$$

$$\Omega =\sum _{(x,y)\in \mathcal{X}}{\mathbf{e}}_y\cdot Q(\mathbf{x}{)}^{\top }$$

$$h\left(\Omega \cdot Q({\mathbf{x}}^{\prime })\right)$$

Full text

In this paper, we introduce a novel layer designed to be used as the output of pre-trained neural networks in the context of classification. Based on Associative Memories, this layer can help design deep neural networks which support incremental learning and that can be (partially) trained in real time on embedded devices. Experiments on the ImageNet dataset and other different domain specific datasets show that it is possible to design more flexible and faster-to-train Neural Networks at the cost of a slight decrease in accuracy.

1 Introduction

During the past decade, deep neural networks, and more specifically Deep Convolutional Neural Networks have been established as the state-of-the-art solution for various problems of Computer Vision such as image classification [19, 31, 13, 33], image segmentation [22, 12, 3] and object tracking [34, 2].

A standard Deep Neural Network relies on millions of trained parameters and thus requires millions of floating point operations in order to compute the output corresponding to a given input. Consequently, the use of deep neural networks for inference in real time tasks requires massive computing power and large amounts of memory. However embedded devices have important limitations in terms of computing power, memory and battery usage, so that deep neural networks are difficult to implement. Many research works have been carried out in order to produce faster deep neural networks architectures at run-time ; new specific architectures have been developed specifically for real-time execution [28] and embedded systems [14, 30]. Quantization [36, 39, 10, 11], and more generally binarization [4, 26, 40], of deep neural networks is the preferred solution for running inference on embedded devices. It permits, at the cost of a little decrease of the accuracy, to replace the computationally intensive floating-point operations by low-bit operations which can be more efficiently implemented, especially on FPGA.

Nevertheless, those research are mostly focused on the efficient execution of the inference on the embedded device and not on the even more complex training procedure which requires going through a large dataset multiple time. As of today, this procedure is generally performed offline using specific hardware such as GPUs or TPUs. Moreover, neural networks trained sequentially using backpropagation algorithm have a high propensity to steeply forget previous tasks when learning new ones regardless of whether they are used for reinforcement learning or supervised learning. This situation, referred to as catastrophic forgetting [7, 8], happens because the weights of the network previously optimized for the first tasks are overwritten in order to correctly achieve the new task. Thus, adding new elements to the dataset (or new classes) can be handled only by restarting the training from scratch.

In order to benefit from the accuracy of deep neural networks without having to train them, a common solution is to rely on transfer learning [24]. In the context of computer vision, transfer learning consists in using deep neural networks pre-trained on a large dataset such as ImageNet [5], Microsoft COCO [21] or Google OpenImages [20], in order to obtain a generic image representation for other tasks. It is then possible to address new classification tasks on a different dataset by using pre-trained models as feature extractors, which may then be fine-tuned and combined with a simple, sometimes incremental, classifier [6, 27, 38, 1]

In this paper, by using Self-Organizing Maps and Sparse Associative Memories, we propose a new Neural Network model meant to be used for classification tasks using transfer learning with pre-trained deep neural networks. The method we introduce comes with the following interests:

• •

It is able to incrementally learn new classes, while achieving competing accuracy with off-the-shelf non-incremental transfer methods

• •

It performs learning with a very limited complexity compared to existing counterparts, making it a competitive solution for embedded devices

• •

It builds on top of well-known models of Associative Memories and Self-Organizing Maps, each with its own set of hyperparameters that can be advantageously tuned in order to adapt to the ad-hoc constraints of a given problem

We experimentally prove these points in Section 4 by performing experiments using competitive vision transfer benchmarks.

2 Self-Organizing Maps and Sparse Associative Memories

2.1 Self-Organizing Maps

2.1.1 Presentation

A Self-Organizing Map (SOM) [18] is a fully connected layer of $N$ neurons that associates a $d$-dimensional input vector $\mathbf{x}$ with an $N$-dimensional output vector $\mathbf{q(x)}$. These neurons are organized on a 2D-grid of $q$ by $r$ units in a way that each neuron but those on the edges has 4 direct neighbors. All these neurons are entirely connected with the $d$ neurons of the previous layer. The weights corresponding to a given neuron $i$ are denoted $\mathbf{w}_{i}$. Figure 1 depicts an example of such an input layer and a map layer.

2.1.2 Inference

When an input vector $\mathbf{x}$ is presented to the map layer, the corresponding output is computed as a vector $q(x)\in\{0,1\}^{N}$ containing a single 1. The coordinate $i^{*}$ which value is 1 is defined as $i^{*}=\arg\min_{i=1}^{N}{dist(\mathbf{w}_{i},x)}$. This $i^{*}$-th neuron is referred to as the Best-Matching Unit (BMU). Note that when the vector $x$ and the vectors $\mathbf{w}_{i}$ all have a unit norm, the dynamics of self-organizing maps is equivalent to:

[TABLE]

where $h$ is a Winner-Takes-All (WTA) operator (all values are put to 0 except for the maximal one, which is put to 1) and $W$ is the matrix which lines are the vectors $\mathbf{w}_{i}$.

2.1.3 Training

In contrast with classic fully-connected layers used in deep neural networks, SOMs are built so that neighbor units contain strong inner dependencies, as explained below. The learning algorithm is performed for a specific number of epoch E and a specific batch size. The parameters $\mathbf{w}_{i}$ are first initialized at randoms. The learning procedure is then performed by iterating the following operations for t from 0 to E:

The training set $\mathcal{X}$ is randomly shuffled. 2. 2.

For each input vector $\mathbf{x}$ in $\mathcal{X}$, the corresponding output vector $\mathbf{y}$ is computed. Denoting $i^{*}$ as the neuron where $\mathbf{y}(i^{*})=1$, we perform the following update of all weights:

[TABLE]

where $A$ is the learning decay function expressed as $A(t)=\alpha T(t)$ with $\alpha$ is the learning rate and $T$ is a function which decreases with $t$ that can be expressed as $T(t)=1-\frac{t}{E}$ or $T(t)=e^{-\frac{t}{E}}$ ; $\Theta$ the neighborhood function which decreases with $t$ and the distance in the grid between neurons $i$ and $i^{\prime}$ defined as $\Theta(t,i,i^{\prime})=e^{-\frac{d_{i,i^{\prime}}^{2}}{2(\theta T(t))^{2}}}$ where $T$ is the decreasing function defined above and $d_{i,i^{\prime}}$ is the distance between the $i$-th and the ${i^{\prime}}$-th neurons in the grid independently of their associated vectors $\mathbf{w}_{i}$ and $\mathbf{w}_{j}$. Note that the latter depends only on the topology of the chosen self-organizing map.

2.1.4 Quantizing with multiple SOMs

A popular way to quantize a vector is to use Product Quantization [15] (PQ). PQ consists in the following: a) splitting the input vectors into $k$ distinct subparts and b) quantizing each part individually and independently from the others. The term “product” comes from the fact that the initial space is divided into a Cartesian product of lower dimensional subspaces.

In this section we propose to use multiple SOMs in order to perform PQ, using one for each subspace. The study is restricted to the case where vectors in each subspace all have the same dimension and where each SOM contains the same number $N$ of neurons, such that the number of anchor vectors in the product (initial) space is $N^{k}$.

Concretely, let us consider $dk$-dimensional input vectors. Our methodology is summarized as follows:

• •

Training:

Initialize $k$ SOMs with input dimension $d$ (indexed from 1 to $k$). They each contain $N$ neurons, 2. 2.

Split training vectors regularly into $k$ $d$-dimensional subvectors each. The first subvector of each train vector is used to train the first SOM, the second subvector of each train vector to train the second SOM, etc.

• •

Quantizing:

Split the input vector $x$ into the $k$ corresponding subvectors, 2. 2.

For each subvector, obtain the corresponding output subvector using the associated SOM, 3. 3.

Concatenate the output subvectors to obtain a $kN$-dimensional binary vector containing exactly $k$ 1s, denoted $Q(x)$.

2.2 Sparse Associative Memories

Sparse Associative Memories [9] (SAMs) are neural networks able to store and retrieve sparse patterns from incomplete inputs. They consist of a neural network made of $p$ distinct groups composed of variable numbers of units bound by binary connections. SAMs are able to store patterns which have exactly one active neuron in each group. Although not optimized for this problem, SAMs are also able to retrieve “close” patterns, meaning that some initially active neurons are changed during the retrieving procedure to find a stored pattern.

The learning procedure is as follows: The connections between the neurons are initialized empty ; then for each pattern to store, the corresponding connections are added to the network. Since connections are binary (they either exist or not), they are not reinforced if shared by more than a single pattern.

The retrieving procedure starts from a partial pattern, meaning that some of its active neurons are initially not activated. Then, an iterative procedure is started. This procedure consists in finding in each group of neurons, the neuron (or the neurons) that has the maximum number of connections with the active neurons. Hopefully, after a few iterations, the network stabilizes to the stored pattern.

3 Proposed model: combining SOMs and SAMs

3.1 Presentation

We propose a new model by combining SAMs with multiple SOMs. The proposed model is a sparse associative memory composed of $p=k+1$ groups, where the $k$ first groups correspond to $k$ self-organizing maps composed of $N$ neurons each, and the last group is the output layer containing $M$ neurons. Since we are only interested in finding the active neuron in the last group, connections between the first $k$ groups are ignored and the retrieving procedure is not iterated.

The proposed model has two hyper-parameters: $k$ the number of SOMs and $N$ the number of neurons on each SOM (as stated before, we assume that each SOM has the same number of neurons).

3.2 Training

The proposed model is expressed as $k$ weight matrices $W$s corresponding to each SOM and one sparse matrix $\Omega$ representing the connections between the $k$ SOMs and the output layer. We denote $\mathcal{X}$ the training set containing pairs ($\mathbf{x}$, $\mathbf{y}$), where $\mathbf{x}$ is a $dk$-dimensional vector and $\mathbf{y}$ is an integer value between 1 and $M$ corresponding to the label. The learning procedure consists in two distinct steps:

Training the multiple SOMs as described in the section 2.1.4 on the training dataset $\mathcal{X}$ in order to compute the weight matrix $W$ associated with each SOM. 2. 2.

The sparse matrix $\Omega$ is made of $M$ lines and $kN$ columns, initially containing only 0s. Two methods are proposed in order to train $\Omega$:

• •

Binary method: the sparse matrix $\Omega$ is a binary matrix containing only 1s and 0s. Then for each couple ($\mathbf{x}$, $\mathbf{y}$) in the training set $\mathcal{X}$, the following equation is obtained:

[TABLE]

where ⊤ is the transpose operator, the $\max$ is applied componentwise, $\mathbf{e}_{\ell}$ is the vector of size $M$ containing only 0s except for one 1 at the $\ell$-th coordinate and $Q$ is the quantization function which uses the $k$ SOMs of the model as defined in the section 2.1.4.

• •

Integer method: the sparse matrix $\Omega$ is an integer matrix containing only positive integer values. Then for each couple ($\mathbf{x}$, $\mathbf{y}$) in the training set $\mathcal{X}$, the following equation is obtained:

[TABLE]

where ⊤ is the transpose operator, $\mathbf{e}_{\ell}$ is the vector of size $M$ containing only 0s except for one 1 at the $\ell$-th coordinate and $Q$ is the quantization function which uses the $k$ SOMs of the model as defined in the section 2.1.4.

For reasons of clarity, we denote $k$x$N$-AL the proposed model composed of $k$ SOMs with $N$ neurons on each SOM if it was trained using the binary method and $k$x$N$-IAL if it was trained using the integer method.

The binary method is the faster one because it takes advantage of the fact that both $\Omega$ and $Q(x)$ are binary variables, implying that the computation of the product of both can be highly optimized during implementation. However, it is important to notice that a lot of information is lost due the binary representation of elements inside the matrix $\Omega$ and the integer method has been designed to mitigate this loss. Although it is no longer possible to take advantage of binary operations to optimize the implementation, integer operations remain faster than floating-point operations.

3.3 Inference

The retrieving procedure is also twofold and does not depend on the method used for the training. By considering an input vector $\mathbf{x}^{\prime}$, the following prediction can be obtained by:

[TABLE]

where $h$ is an activation function, the Winner-Takes-All (WTA) function is the most commonly used and $Q$ is the quantization function which uses the $k$ SOMs of the model as defined in the section 2.1.4.

When used this way, the sparse associative memory basically emulates a majority vote among the multiple self-organizing maps.

3.4 Proposed model used as classifier

3.4.1 Presentation

The model composed of multiple self-organizing maps and one sparse associative memory proposed in the previous section has been designed to be used as a new neural network classifying layer for a conventional Deep Neural Network. The proposed model is a neural network classifier that relies on Transfer Learning and it has to be combined with a feature extractor. In general, the feature extractor is a Convolutional Neural Network pre-trained on a universal dataset, such as ImageNet for image classification, which classification layer has been removed.

3.4.2 Training

We denote $\mathcal{U}$ the universal training dataset used to train the feature extractor. $\mathcal{U}$ contains pairs ($\mathbf{x}$, $\mathbf{y}$), where $\mathbf{x}$ is a training vector and $\mathbf{y}$ is the corresponding label, and $\mathcal{U^{\prime}}$ the dataset containing pairs ($\mathbf{x^{\prime}}$, $\mathbf{y}$) where $\mathbf{x^{\prime}}$ is a k-dimensional vector obtained as the output when $\mathbf{x}$ is passed as the input to the feature extractor. Similarly, we denote $\mathcal{T}$ the domain specific training set and $\mathcal{T^{\prime}}$.

As described in section 3.2, the first step is to train the SOMs of the proposed model on the universal dataset $\mathcal{U^{\prime}}$. The second step consists in training from scratch the matrix $\Omega$ using the domain specific training set $\mathcal{T^{\prime}}$. The first step, training the SOMs, is the most computationally expensive task and has to be done ahead ; while the second step, the training on the domain specific dataset, takes fully advantage of the fast incremental learning algorithm described above and has been designed to be done in real-time on the edge.

3.4.3 Application

The strengths of this learning procedure is that it is simple and fast: once the feature extractor and the self-organizing maps have been trained on the universal dataset, each training element of the specific dataset has to be processed only one time in order to be learned and the procedure is limited to two sparse matrix products and non-linear functions if the input of the model is a unit vector. Moreover, the learning of each element is independent, and thus it is possible to learn new elements in parallel and incrementally. This means that contrary to deep neural networks which rely on the gradient decent algorithm, it is not required to restart the training from scratch in order to learn new classes or new elements for a previously learned class. Therefore in the situation where the proposed model has already been trained on several classes and has to learn a new one, it is just required to process the elements of the new class. While for a standard deep neural network trained using gradient decent algorithm, in order to avoid catastrophic forgetting it is required to restart the learning from scratch and to process again the elements of the previously learned classes in addition to the elements of the new class.

4 Experiments

4.1 Protocol

In the following subsections, the impact of the different hyper-parameters of the proposed model is evaluated and the proposed model is compared with other methods on several small domain-specific datasets.

In order to be able to express the complete layer as matrix products, inputs of the proposed model are normed: this does not significantly impact the accuracy of the model in this case.

Because the accuracy and the time required for the training may slightly vary, every experiment has been done thirty times. The mean and the standard deviation are reported for each measure. Measures of the training time only consider the time required to train the classifiers and do not take the time required to extract the features into account.

4.1.1 Competing methods

In order to evaluate our method, the proposed model is compared with different classifiers: a brute-force K-Nearest Neighbor classifier, a Support Vector Machine and a simple deep neural network classifier (denoted DNN classifier). The C-SVM with a Gaussian kernel has been used and it has been trained using a One-vs-Rest policy. The deep neural network is composed of three densely connected layers and it has been trained using three different optimizers : stochastic gradient descent, stochastic gradient descent combined with Cyclical Learning rates [32] and Adam [17]. All classifiers, just like the proposed model, are trained using transfer learning with the same feature extractor. The feature extractor used in all the experiments below is a VGG-16 model trained on ILSVRC dataset [29] whose dense classification layer has been removed, thus the feature vector is a 4096-dimensional floating point vector. In order to compare the four solutions, it has been decided to not use augmentation on training data or fine-tuning of previous layers of the pre-trained VGG-16 model: the use of these techniques will improve the performance of every method.

4.1.2 Datasets

The proposed model has been compared with the other classifiers on several domain specific dataset: 102 Category Flower Dataset [23] (denoted by Flower102), the Indoor Scene Recognition Dataset [25] (denoted by Indoor67), the Caltech-UCSD Birds 200 dataset [35] (denoted by CUB200), the Stanford Dogs Dataset [16] (denoted by Dog120) and the Stanford 40 Actions Dataset [37] (denoted by Stanford40). These datasets contain highly similar images divided into a lot of classes containing only few images each:

• •

Flower102 dataset contains images of 102 different flower species : the training set is composed of 2040 images and the test set is composed of 6149 images.

• •

Indoor67 dataset contains images of 67 different indoor places : the training set is composed of 5360 images and the test set is composed of 1340 images.

• •

CUB200 dataset contains images of 200 different bird species : the training set is composed of 3000 elements and the test set is composed of 3033 elements.

• •

Dog120 dataset contains images of 120 different dog species : the training set is composed of 12000 elements and the test set is composed of 8580 elements.

• •

Stanford40 dataset contains images of 40 distinct type of actions performed by humans : the training set is composed of 4000 elements and the test set is composed of 5532 elements.

4.2 Impact of hyper-parameters

The first series of experiments consist in comparing the accuracy of the proposed model on the classification of the Stanford Dogs Dataset depending on the two hyper-parameters of the proposed model: $k$ the number of self-organizing maps and $N$ the number of neuron in each self-organizing map.

Figure 4 and Figure 4 compare the Top-5 accuracy of different versions of the proposed model on the classification of Dog120. Figure 4 compares versions of the proposed model which each has a different number of SOMs and Figure 4 compares versions of the proposed model which each has a different number of neurons in each SOM. It appears that, both the increase of the number of SOMs composing the model and the increase of the number of neurons in each SOM will increase the accuracy of the model. This is tied to the fact that they will reduce the loss of information induced by the quantization. However, the computational cost of both the training procedure and the inference procedure of the proposed model is directly correlated to the total number of neurons in the model: the sum of the number of neurons in each SOM. Thus, increasing the number of SOMs or the number of neurons per SOM will also increase the cost. It should be noted that increasing the number of neurons is not necessarily synonymous with increasing the accuracy of the model. As shown in Figure 4, the 32x256-AL is outperformed by the 64x100-AL even though the latter has 28% less neurons than the first one. In general, for a fixed number of neurons, the model using the largest number of SOMs should be preferred.

Figure 6 and Figure 6 compare the original proposed model with the improved version which uses integer values instead of binary values on two different classification tasks. The binary model is more accurate than the integer one on the Dog120 dataset while the integer model is more accurate on the ILSVRC dataset. In fact the binary model is more efficient for datasets containing few images per classes whereas the integer model is more efficient on larger datasets containing more images per classes.

4.3 Comparison with other methods

Figure 8 compares the accuracy of the proposed model with the accuracy of the deep neural network classifier trained on the Dog120 dataset using stochastic gradient descent depending on the number of learned classes. Both classifier have been trained on all the considered classes at once: this means that each time a new class is added to the training set, the learning procedure is restarted from scratch and the classifier has to learn again the previously learned classes in addition to the new class. It appears that both models have nearly the same accuracy: they have the same Top-5 accuracy and the Top-1 accuracy of proposed model is slightly inferior to the one of the deep neural network classifier. In the case where the number of learned classes equals 120, the loss in accuracy is about 2.7%: the Top-1 accuracy of the proposed model is 71.9% and the the Top-1 accuracy of the deep neural network classifier is 74.6%.

However, as explained in the section 3.4.3, contrary to the deep neural network classifier, the proposed model supports incremental learning. Figure 8 compares the accuracy of the proposed model with the accuracy of the deep neural network classifier trained on the Dog120 dataset using stochastic gradient descent depending on the number of learned classes. Nevertheless, in this case both classifiers have been trained sequentially: this means that each time a new class is added to the training set, the classifier is only trained on the new elements. As expected, the accuracy of the proposed model is exactly the same because of the learning algorithm used: it does not change if the proposed model is trained on all the dataset at once or sequentially. However, the accuracy of the deep neural network classifier crashed and dropped to 0.66% for Top-1 accuracy and 4.55% for Top-5 accuracy in the case where the number of learned classes equals 120 . This is due to catastrophic forgetting: since the deep neural network classifier was trained sequentially, each time it learns a new class it also inevitably forgets all the previously learned classes. Therefore, the deep neural network classifier always predicts the last learned class independently of the input image.

As a result, the proposed model is more flexible than the deep neural network classifier because it supports incremental learning at the cost of the slight decrease of accuracy. The proposed model can be more easily retrained and adapted to a new situation compared to the deep neural network classifier. For example, if the proposed model has already been trained on the first 119 classes of the Dog120 dataset and has to learn the last one, it is just required to process the elements of the new class, while for the deep neural network classifier, it is required to restart the learning from scratch and to process the elements of all the 120 classes of the dataset in order to avoid catastrophic forgetting. In this case, it will be up to approximately 2600 times faster for the proposed model to learn the new class compared to the deep neural network classifier, as shown in Table 3.

The proposed model has also been compared with the other classifiers on several additional domain specific dataset:

Table 1 shows that the proposed model is slightly inferior to the others methods in terms of performance on the tested datasets except for the CUB200 dataset. However, due to the learning algorithm used, it is considerably faster to train as shown in Table 2. In order to moderate these results, it may be possible to decrease a bit the training time of the other models by using a more aggressive early stop function and different hyper-parameters but the order of magnitude will remain similar: generally the proposed model is 10 times faster to train than the others when trained on the whole dataset at once. However, as explained above, if it is required to only learn a new class, both the deep neural network classifier and the SVM have to be retrained from scratch and thus, the proposed model is exceptionally faster to train, as shown in Table 3.

5 Conclusion

We introduced a new classifier model primarily composed of Self-Organizing Maps and Sparse Associative Memories. By combining the proposed layer with a pre-trained Deep Neural Network, it is possible to design flexible deep neural networks with a considerably faster learning algorithm as well as the support of incremental learning at the cost of a slight decrease of the accuracy. Using the proposed model can help the development and the deployment of new intelligent embedded devices which can learn new elements in real time and adapt themselves to their environment without the help of an external agent responsible for the execution of the training algorithm.