A Semi-Supervised Self-Organizing Map with Adaptive Local Thresholds

TL;DR

This paper introduces ALTSS-SOM, a semi-supervised learning method based on Self-Organizing Maps that adaptively adjusts to local data variance, improving classification and clustering performance with robustness to parameter settings.

Contribution

It proposes a novel ALTSS-SOM model that dynamically switches learning modes and automatically adapts to local data variance, enhancing semi-supervised clustering and classification.

Findings

ALTSS-SOM outperforms other semi-supervised methods in classification tasks.

ALTSS-SOM surpasses pure clustering methods when labels are absent.

The method is less sensitive to parameter variations than previous approaches.

Abstract

In the recent years, there is a growing interest in semi-supervised learning, since, in many learning tasks, there is a plentiful supply of unlabeled data, but insufficient labeled ones. Hence, Semi-Supervised learning models can benefit from both types of data to improve the obtained performance. Also, it is important to develop methods that are easy to parameterize in a way that is robust to the different characteristics of the data at hand. This article presents a new method based on Self-Organizing Map (SOM) for clustering and classification, called Adaptive Local Thresholds Semi-Supervised Self-Organizing Map (ALTSS-SOM). It can dynamically switch between two forms of learning at training time, according to the availability of labels, as in previous models, and can automatically adjust itself to the local variance observed in each data cluster. The results show that the ALTSS-SOM surpass the performance of other semi-supervised methods in terms of classification, and other pure clustering methods when there are no labels available, being also less sensitive than previous methods to the parameters values.

Tables

Table 1. TABLE I: Parameter Ranges for ALTSS-SOM

Parameters	min	max
Lowest cluster percentage (lp)	0.001	0.002
Relevance rate ($\beta$)	0.90	0.95
Max competitions ($age\_wins$)	$1\times S^{\ast }$	$200\times S^{\ast }$
Winner learning rate ($e_b$)	0.001	0.2
Neighbors learning rate ($e_n$)	$0.002\times e_b$	$1\times e_b$
Relevance smoothness ($ϵ\beta$)	0.01	0.1
Connection threshold ($minwd$)	0	0.5
Number of epochs ($epochs$)	1	100

Table 2. TABLE II: CE Results for Real-World Datasets. Best results for each dataset are shown in bold

CE	Breast	Diabetes	Glass	Liver	Pendigits	Shape	Vowel	Average	STD
DOC	0.763	0.654	0.439	0.580	0.566	0.419	0.142	0.509	0.201
PROCLUS	0.702	0.647	0.528	0.565	0.615	0.706	0.253	0.574	0.156
LARFDSSOM	0.763	0.727	0.575	0.580	0.737	0.719	0.317	0.631	0.158
ALT-SSSOM	0.763	0.697	0.575	0.603	0.741	0.738	0.319	0.633	0.156

Equations

$$s_1(\mathbf{x})=\text{arg}\underset{j}{\max }[ac(D_{\omega }(\mathbf{x},{\mathbf{c}}_j),{\mathbf{\omega }}_j)].$$

$$ac(D_{\omega }(\mathbf{x},{\mathbf{c}}_j),{\mathbf{\omega }}_{\mathbf{j}})=\frac{\sum _{i=1}^m{\omega }_{ji}}{\sum _{i=1}^m{\omega }_{ji}+D_{\omega }(\mathbf{x},{\mathbf{c}}_j)+ϵ},$$

$$D_{\omega }(\mathbf{x},{\mathbf{c}}_j)=\sqrt{\sum _{i=1}^m{\omega }_{ji}{(x_i-w_{ji})}^2}.$$

$${\mathbf{\delta }}_j(n+1)=\beta {\mathbf{\delta }}_j(n)+(1-\beta )(|\mathbf{x}-{\mathbf{c}}_j(n)|),$$

$${\hat{\mathbf{\delta }}}_j(n+1)=\frac{{\mathbf{\delta }}_j(n)}{1-{\beta }^{t_j}}$$

$$\small\omega_{ji}=\begin{cases}\frac{1}{1\text{ + exp}\Big{(}\frac{\hat{\delta}_{ji\text{mean}}-\hat{\delta}_{ji}}{s(\hat{\delta}_{ji\text{max}}-\hat{\delta}_{ji\text{min}})}\Big{)}}&\text{if }\hat{\delta}_{ji\text{min}}\neq\hat{\delta}_{ji\text{max}}\\ 1&\text{otherwise},\end{cases}$$

$$\text{Var}({\hat{\mathbf{\delta }}}_j,{\mathbf{\omega }}_j)=\frac{{\hat{\mathbf{\delta }}}_j}{{\mathbf{\omega }}_j}$$

$$A(\mathbf{x},{\mathbf{c}}_j,{\mathbf{v}}_j)=\{\begin{array}{ll}True,& {\mathbf{x}}_i\in \left]{\mathbf{c}}_{ji}\pm {\mathbf{v}}_{ji}\right[,\\ & \forall {\text{ c}}_{ji}\in {\mathbf{c}}_j\text{, }{\mathbf{x}}_i\in \mathbf{x}\text{, and }{\mathbf{v}}_{ji}\in {\mathbf{v}}_j\\ False,& \text{otherwise,}\end{array}$$

$${\mathbf{c}}_j(n+1)={\mathbf{c}}_j(n)+e(\mathbf{x}-{\mathbf{c}}_j(n)),$$

Full text

I Introduction

Over the last few years, the use of machine-learning technology has driven many aspects of modern society. Recent research on Artificial Neural Networks with supervised learning has shown great advances. It is the most common form of machine learning [1]. It is not unusual to see on the news several practical applications, in diverse areas [1, 2]. A key to the success of supervised learning, especially, deep supervised learning, is the availability of sufficiently large amounts of labeled training data. Unfortunately, creating such properly labeled data with enough examples for each class is not easy. As a result, the use of supervised learning methods became impractical in many applications such as in the medical field, where it is extremely difficult and expensive to obtain balanced labeled data.

On the other hand, due to the advances in technology that have produced datasets of increasing size, in terms of the number of samples and features, unlabeled data usually can be easily obtained. Therefore, it is of great importance to put forward methods that can combine both types of data in order to benefit from the information they can provide, each of them in their way [3]. An approach typically applied in such scenario is Semi-Supervised Learning (SSL). It is a halfway between supervised and unsupervised learning and can be used to both clustering and classification tasks [3, 4].

We point out that prototype-based methods have been successfully applied for both tasks. Methods based on Self-Organizing Maps [5, 6, 2] and K-Means [7] can be highlighted as examples, as well as deep learning techniques [8, 9, 10]. The Self-Organizing Map (SOM) is an unsupervised learning method, frequently applied for clustering, while Learning Vector Quantization (LVQ) [5], its supervised counterpart that shares many similarities, is normally used for classification. They both were proposed by Kohonen, and since then, various modifications have been introduced, including semi-supervised versions [2], to deal with more challenging problems.

Recent SOM-based methods employ a threshold defining the minimum level of activation for an input pattern to be considered associated with a cluster prototype. This threshold level is a parameter of the model which is shared by all prototypes [6, 2], thus, the regions that a prototype can represent are not learned at all, or they are normally estimated using supervised approaches, as in [11].

In this context, the main idea of this paper is to introduce the concept of local adaptive thresholds through the use of the local variances observed by the prototypes for each dimension. Such variances are calculated using a bias-corrected moving average with an exponentially weighted decay rate [12]. This concept was derived from the idea of rejection options, early introduced by Chow [13]. It is related to the conditions of taking a classification or recognition decision for a particular point or a data region. In the case of SOM-based methods, these decisions are associated with the nodes in the map (i.e., when they must accept an input pattern to be part of its representation pool). Such rejection options define the first step towards an adaptation of the model complexity tailored to data regions with a high degree of uncertainty [14]. So far, most models that use rejection options deals with just a single threshold, as well as most of them can handle only with binary classification [11].

In this article, we propose a new model called ALTSS-SOM, which is an extension of Semi-Supervised Self-Organizing Map (SS-SOM), created by introducing important modifications to incorporate the ability to estimate local rejection options as a function of both local variance and relevance of the input dimensions for each node in the map. To evaluate ALTSS-SOM, we compared it with other semi-supervised approaches that do not use adaptive reject options. We also compare ALTSS-SOM with its predecessor that used a parameter to define the threshold region to make pattern rejection decisions. Also, once we introduce an entirely new learning procedure, it becomes necessary to compare ALTSS-SOM not only regarding the classification rate but also considering the clustering rate. It is done by taking into consideration the methods that provided the ideas for the development of the proposed method as well as other conventional methods in the literature. Finally, as our parameter sensitivity analysis shows, the sensibility of the model to the parameters was significantly decreased in comparison with the previous version.

The rest of this article is organizing as follows: Section II presents a short review of the background related to the areas where this paper is inserted. Section III introduces related work in the literature. Section IV describes in detail the proposed method. Section V presents the experimental setup, methodology, the obtained results, and comparisons. Finally, Section VI discusses the obtained results and draws the conclusion of this paper, as well as indicates future directions.

II Background

High-dimensional data poses different challenges for clustering tasks. In particular, similarity measures used in traditional clustering techniques may become meaningless due to the curse of dimensionality [15]. In this context, subspace clustering and projected clustering appear as common choices. They aim to determine clusters in subspaces of the input dimensions. This task involves not only the clustering itself but also identifying relevant subsets in the input dimensions for each cluster [16]. One way to achieve this is by applying local relevances to the input dimensions.

Moreover, as in [2], this paper introduces a model that is able not only to cluster but also classify samples. In this context, we aim to introduce the concept of reject options. According to Chow [13], uncertainties and noise inherent in any pattern recognition task result in a certain amount of unavoidable errors. Uncertainty normally has two reasons: points being outliers or located in ambiguous regions [17]. The option to reject is introduced to avoid an excessive misrecognition rate by converting errors into rejection.

We derive this idea to consider local reject options based on the variances estimated for each node in the map (discussed in more details further in the Section IV). The main idea is to give nodes the ability to reject an input pattern x if it is outside a region in the space defined from the estimated variances in each dimension around the centroid of each node. This will result in an adaptive local thresholding approach, similar to the one found in [18, 19]. However, this variance based approach provides a threshold adjusted for each dimension of each node in the map during the semi-supervised learning process.

Finally, it is important mentioning that SSL perfectly fits for all of the referred problems and techniques. Because of that, considering the growing interest in semi-supervised learning in the past years, such combined approaches may come to arise more often. It is also worth pointing out that such interest for SSL is growing in the machine-learning [7, 20, 2] alongside in the deep learning context [21, 22].

III Related Work

This section briefly summarizes related works in diverse contexts. First, in the purely semi-supervised machine-learning context, we highlight the K-means based methods. K-means is one of the most popular and simple clustering algorithms that was proposed over 50 years ago, but it is still widely used in diverse applications [23, 24].

Continuing talking about prototype-based methods, the SS-SOM [2] appears as the inspiration to the model presented in this paper. SS-SOM is a semi-supervised method that essentially inherits characteristics from its predecessor, Local Adaptive Receptive Field Dimension Selective Self-organizing Map (LARFDSSOM) [6], but also introduces elements of supervision (i.e. supervised learning) to create a hybrid environment where SSL could be applied. The LARFDSSOM [6] uses a time-varying structure, a neighborhood defined by connecting nodes that have similar subspaces of the input dimensions, and a local receptive field that is adjusted for each node as a function of its local variance. Both LARFDSSOM and SS-SOM were developed to deal with high-dimensional data, considering different learning contexts. The latter carries the ability to perform not only clustering tasks but also classification. Note that they works exactly as the same when there is no labeled sample available.

Label Propagation methods [7] are another approach for SSL. Essentially, they operate on proximity graphs or connected structures to spread and propagate class information to nearby nodes according to a similarity matrix. It is based on the assumption that nearby entities should belong to the same class, in contrast to far away entities. Label Spreading [20] methods are very similar. The difference consists of modifications to the similarity matrix. The first uses the raw similarity matrix constructed from the data with no changes, whereas the latter minimizes a loss function that has regularization properties allowing it to be often better regarding robustness to noise.

Furthermore, in the literature, some state of the art strategies for rejection option can be listed [14]. On considering both local and global rejection, with the latter being the most common form, they can be divided into three distinct categories [11]: 1) probabilistic approaches; 2) turning deterministic measures into probabilities, and 3) deterministic approaches. Further, some adaptive local thresholding techniques are also found, like as in [18].

Moreover, it is possible to see in [16] a review of methods that work well for problems of clustering in high-dimensional data. On considering this, and also the detailed comparison of subspace and projected clustering methods perform by [6], we highlight PROjected CLUStering algorithm (PROCLUS) [25], Densitive-based Optimal projective Clustering (DOC) [26] and the LARFDSSOM models due their good performances. DOC [26] is a cell-based method that searches for sets of grid cells containing more than a certain number of objects by using a Monte Carlo based approach that computes, with high probability, a good approximation of an optimal projective cluster. PROCLUS [25] is a clustering-oriented algorithm that aims to find clusters in small projected subspaces by optimizing an objective function of the entire set of clusters, such as the number of clusters, average dimensionality, or other statistical properties.

IV Proposed Method

ALTSS-SOM 111Available at: https://github.com/phbraga/alt-sssom is a SOM with Adaptive Local Thresholds [18] based on SS-SOM. Hence, being based on SS-SOM, ALTSS-SOM can also learn in a supervised or unsupervised way depending on the availability of labels, and maintains the general characteristics of its predecessors. However, it introduces new supervised and unsupervised behaviors to allow a better usage, and consequently a better understanding of the data statistics. By doing this, ALTSS-SOM aims at overcoming the problems presented by SS-SOM, such as the high sensitivity to the parameters, and the low sample efficiency. Additionally, with the proper changes, ALTSS-SOM targets achieving better results for both classification and clustering tasks.

Therefore, the parameterized activation threshold ($a_{t}$) used in both previous methods is replaced by an adaptive thresholding technique that takes into account the local variance to provide the model the ability to learn the receptive field of each node. The objective is to estimate optimal local regions in the space with respect to the distribuition of the input patterns $\boldsymbol{x}$ for each node in the map. To do so, inspired by the Adam algorithm, a method for efficient stochastic optimization that only requires first-order gradients with little memory requirement [12], ALTSS-SOM updates exponential moving averages of the distances between each input pattern $\boldsymbol{x}$ and the centroid of the nodes for each dimension (the vector $\boldsymbol{\delta}_{j}$ in the algorithms). In SS-SOM and LARFDSSOM, this estimate was done by using not only $\beta$ but also the learning rate $e$. However, ALTSS-SOM modified this approach to use solely the parameter $\beta\in\left[0,1\right)$ for controling the exponential decay rate of the moving averages.

The moving averages themselves are estimates of the first moment (the mean) of the distances between the input patterns and the centroids of the nodes. Because of that, such means can be used as estimates of the uncentered variance of the nodes in each dimension. However, these moving averages are initialized as vectors of zeros, leading to moment estimates that are biased towards zero, especially during the initial steps, and when the decay rate is small (close to 1) [12]. Still, according to [12], this initialization bias can be counteracted, resulting in the bias-corrected estimate $\boldsymbol{\widehat{\delta_{j}}}$.

The moving averages themselves are estimates of the first moment (the mean) of the distances between the input patterns and the centroids of the nodes. Because of that, such means can be used as estimates of the uncentered variance of the nodes in each dimension. However, these moving averages are initialized as vectors of zeros, leading to moment estimates that are biased towards zero, especially during the initial steps, and when the decay rate is small (close to 1) [12]. Still, according to [12], this initialization bias can be counteracted, resulting in a bias-corrected estimate $\hat{\boldsymbol{\delta}}_{j}$. During the learning process, this bias-corrected estimate $\hat{\boldsymbol{\delta}}_{j}$, together with the relevance vector $\boldsymbol{\omega}_{j}$ can be used as reject options [13], determining whether or not an input pattern is in the receptive field of a winner node.

The overall operation of the map comprises three phases: 1) organization (Algorithm 1); 2) convergence; and 3) clustering or classification.

In the organization phase, the network is initialized, and the nodes start to compete to form clusters of randomly chosen input patterns. The first node of the map is created at the same position of the first input pattern. As in SS-SOM, there are two distinct ways to define the winner of a competition, to decide when a new node must be inserted and when the nodes need to be updated. However, in ALTSS-SOM, before a node is updated, it is necessary to decide if it will affect the whole node structure or just the weighted averages and the relevance vectors. If the input pattern class label is provided, it will be done in the supervised mode (Section IV-G), otherwise, in the unsupervised mode (Section IV-F).

The neighborhood of ALTSS-SOM is defined as the same as in SS-SOM. Nonetheless, it defines the nodes that will be adjusted together with the winner, thus outlining the cooperation step. The competition and cooperation steps are repeated for a limited number of epochs, and during this process, some nodes are removed periodically, conforming to a defined removal rule that a node must win at least for a minimum number of patterns to continue in the map, as in SS-SOM.

The convergence phase starts right after the organization process. In this phase, the nodes are also updated and removed when required, like in the way conducted in the first phase but with a slight difference: there is no insertion of new nodes. Finally, when the convergence phase finishes, the map clusters and classifies input patterns. At this stage, as in the SS-SOM, there are three possible scenarios: 1) all of the nodes have a defined representing class; 2) a mixed scenario, with some nodes labeled and other not; and 3) none of the nodes labeled. The first scenario will allow both classification and clustering tasks to be executed straightforwardly. The second will add one more step to the process because if the most activated node does not have a defined class, the algorithm continues trying to find a next highly activated node with a defined class. The last scenario only provides the ability to cluster.

IV-A Structure of the Nodes

In ALTSS-SOM, each node j in the map represents a cluster and is associated with four m-dimensional vectors, where m is the number of input dimensions: The first three vectors, $\boldsymbol{c}_{j}$, $\boldsymbol{\omega}_{j}$, and $\boldsymbol{\delta}_{j}$, are the same as defined in [2], , where $\textbf{c}_{j}=\{c_{ji},i=1,\cdot\cdot\cdot,m\}$ is the center vector; $\boldsymbol{\omega}_{j}=\{\omega_{ji},i=1,\cdot\cdot\cdot,m\}$ is the relevance vector; $\boldsymbol{\delta}_{j}=\{\delta_{ji},i=1,\cdot\cdot\cdot,m\}$ is the distance vector that stores moving averages of the observed distance between the input patterns x and the center vector $|\textbf{x}-\textbf{c}_{j}(n)|$ for each dimension. Note, however, that $\boldsymbol{\delta}$ in SS-SOM and ALTSS-SOM can be seen as the biased first moment estimate. Because of that, ALTSS-SOM introduces a fourth vector, $\hat{\boldsymbol{\delta}}_{j}=\{\hat{\delta}_{ji},i=1,\cdot\cdot\cdot,m\}$, which is the bias-corrected first moment estimate that the algorithm computes to counteract the bias towards zero of $\boldsymbol{\delta}_{j}$, specifically at the initial steps. The $\hat{\boldsymbol{\delta}}_{j}$ vector is used to compute the relevance vector $\boldsymbol{\omega}_{j}$, and both of them are used to approximate the variance of each node, taking into account how significant each dimension is. Such variance is used to define local reject options during the learning process every time that a new input pattern is presented to the map.

IV-B Competition

ALTSS-SOM tries to choose the winner of a competition as the most activated node given an input pattern $\boldsymbol{x}$, except in certain cases that will be discussed in Section IV-G, when the label is available. In ALTSS-SOM, likewise in SS-SOM, the most activated node s($\boldsymbol{x}$) is defined as per Eq. 1:

[TABLE]

where $\boldsymbol{\omega_{j}}$ is the relevance vector of the node j and $ac(D_{\omega}(\boldsymbol{x},\boldsymbol{c}_{j})$ is the activation function.

As in SS-SOM, the activation function in ALTSS-SOM is calculated according to a radial basis function with the receptive field adjusted as a function of its relevance vector $\boldsymbol{\omega}_{j}$, as shown in Eq. 2:

[TABLE]

where $\epsilon$ is a small float number added to avoid division by zero and $D_{\omega}(\boldsymbol{x},\boldsymbol{c}_{j})$ is the weighted distance function:

[TABLE]

IV-C Estimating bias-corrected moving averages

In ALTSS-SOM, the procedure that updates the distance vectors, as well as the relevance vectors, is shown in the Algorithm 2.

The distance vectors are initialized as a vector of zeros and updated through a moving average of the observed distance between the input pattern and the current center vector $\boldsymbol{c}_{j}$, as per Eq. 4:

[TABLE]

where $\beta\in\left]0,1\right]$ is the parameter that controls the rate of change of the moving average (i.e., the exponential decay rate), and $|\boldsymbol{x}-\boldsymbol{c}_{j}(n)|$ denotes the absolute value applied to the elements of the vectors.

In order to correct the bias towards zero of $\boldsymbol{\delta}_{j}$ at the initial timesteps, caused by initializing the moving averages with zeros, as in the Adam algorithm [12], ALTSS-SOM divides it by the term $\left(1-\beta^{t_{j}}\right)$, where $\textit{t}_{j}$ indicates the current timestep of each node j. In sum, the bias-corrected moving averages vectors are updated at every node timestep according to the Eq. 5:

[TABLE]

To obtain accurate information about the relevance of each dimension for a given node, an update of the relevance vectors must follow every moving averages update. It is calculated by an inverse logistic function of the bias-corrected estimated distances $\hat{\boldsymbol{\delta}}_{ji}$, as follows in Eq. 6.

[TABLE]

where $\hat{\delta}_{ji\text{max}},\hat{\delta}_{ji\text{min}}$ and $\hat{\delta}_{ji\text{mean}}$ are respectively the maximum, the minimum and, the mean of the components of the bias-corrected moving average vector $\hat{\boldsymbol{\delta}}_{j}$, and the the parameter $s>0$ controls the slope of the logistic function [6]. This function is pretty similar to the one used in SS-SOM, however, instead of using $\boldsymbol{\delta}_{j}$, the ALTSS-SOM replaces it by $\hat{\boldsymbol{\delta}}_{j}$ in order to get a more accurate and unbiased moving average value.

IV-D Local Thresholds

The distance vectors $\hat{\boldsymbol{\delta}}$ represent the corrected moving average of the observed distances between the input patterns $\boldsymbol{x}$ and center vectors $\boldsymbol{c}$ for each node j in the map. As a result, they can be considered as the variances of the nodes, as stated before.

In addition, the $\boldsymbol{\omega}$ vectors express how each of the dimensions is important for each node, which indicates the subspaces of the input dimensions of a given dataset. This information corroborates with the definition of a local threshold, together with the estimated variance in the form of the $\hat{\boldsymbol{\delta}}$ vectors.

Combining them, it is possible to define a local region around each node center $\boldsymbol{c}_{j}$ to act like a reject option for some input patterns. If only the variances were used, some unimportant dimensions with a low variance could misguide the process when a similar input pattern $\boldsymbol{x}$ is outside the acceptance region of a node j, but only in dimensions that are not relevant to it. Therefore, a flexible variance is defined to act as a local threshold and rejection option to mitigate such problems:

[TABLE]

When a dimension has a high relevance to the node, it will not impact its variance value. However, when a dimension has a small relevance, ALTSS-SOM will relax the constraints to allow a better definition of subspaces. Therefore, the general acceptance rule is defined by Eq. 8, where the idea is to approximate to an optimal rule.

[TABLE]

where $\boldsymbol{x}$, $\boldsymbol{c}_{j}$ are respectively the input pattern, and the center vector, and $\boldsymbol{v}_{j}$ = Var$(\hat{\boldsymbol{\delta}}_{j},\boldsymbol{\omega}_{j})$ is the relaxed variance vector as per Eq. 7.

IV-E Node Update

As in LARFDSSOM, ALTSS-SOM updates the winner node and its neighbors using two distinct learning rates, $e_{b}$, and $e_{n}$, respectively. The Algorithm 3 shows how the whole update occurs.

The relevances and weighted moving averages are updated as shown in Section IV-C, and the centroid vector $\boldsymbol{c}_{j}$, given a learning rate lr, is updated as follows:

[TABLE]

IV-F Unsupervised Mode

Given an unlabeled input pattern, the most activated node is considered as the winner, disregarding its class labels. In this sense, ALTSS-SOM verifies if the condition expressed by Eq. 8 is satisfied. If so, the winner and its neighbors are updated towards the input pattern. Otherwise, a new node is inserted into the map at the input pattern position. However, since $s_{1}$ is the original winner, it will improve its knowledge about the region where it is located by updating its moving averages and relevances, but not its center. This mechanism provides the nodes the ability to learn about the region they are inserted in. An additional case is handled when the map has reached the maximum number of nodes. In this case, aiming at not losing the information that the input pattern can provide, as in previous models, and improving sample efficiency, ALTSS-SOM updates the moving average and the relevance vectors of the winning node. Algorithm 4 illustrates this procedure.

IV-G Supervised Mode

Algorithm 5 shows how this supervised procedure is conducted. In this procedure, unlike the unsupervised mode, the labels are taken into account when looking for a winner. If the most activated node $s_{1}$ has the same class of the input pattern or a not defined class, a very similar approach to the unsupervised mode is applied. The difference is directly related to the fact that is necessary to set $s_{1}$ class as the same class of the given input pattern $\boldsymbol{x}$, as well as to update its connections. Otherwise, the ALTSS-SOM tries to find a new winner with the same class of the input pattern $\boldsymbol{x}$ or a not yet defined class.

If some new node takes the place of $s_{1}$ as a new winner $s_{2}$, the acceptance criteria expressed by the Eq. 8 is verified. If so, and the map is not full, the new winner and its neighbors are updated. Otherwise, only the moving averages and relevance vector of $s_{2}$ are updated in order to give the same chance received by $s_{1}$ to improve its knowledge about the surrounding area.

If there are no new nodes to replace $s_{1}$ as a new winner, and the map is not full, the $s_{1}$ node is duplicated, preserving the moving averages vectors, the centroid vector as well as the relevance vector. However, the class of this new duplicated node is set to the same as the input pattern. The other parameters are set as usual. If none of the above conditions are fulfilled, the ALTSS-SOM solely updates the moving averages and relevance vector of the first defined winner $s_{1}$.

IV-H Node Removal

In ALTSS-SOM, as in SS-SOM and LARFDSSOM, each node j stores a variable $wins_{j}$ that accounts for the number of nodes victories since the last reset. Whenever nwins reaches the age_wins value, a reset occurs. It implies to the moment when nodes that did not win at least the minimum percentage of the competition $lp\times age\_wins$ are removed from the map. After a reset, the number of victories of the remaining nodes is reset to zero. Finally, to avoid the removal of recently created nodes, when a new node is inserted, its number of wins is set to $lp\times nwins$, where nwins indicates the number of competitions that have occurred since the last reset.

IV-I Parameters Summary

ALTSS-SOM removes two parameters from its predecessor, SS-SOM. First, the parameter $a_{t}$, that has a great impact on the results as shown by [6]. It was replaced by the adaptive local threshold technique introduced by ALTSS-SOM (Section IV-D) that can define and learn the space region that a node can represent during the training. Second, the parameter $e_{w}$ was also removed due to its irrelevance in the learning process after removing $a_{t}$, i.e., ALTSS-SOM has nine parameters to be set up. More precisely, a sensitivity analysis revealed that there is no parameter with a high impact on the results anymore. This method seeks to establish a good level of self-adjustment, in a way that we can keep the parameters values fixed inside predefined ranges.

V Experiments

The experiments are divided into three parts. First, in order to evaluate the classification rate of ALTSS-SOM, we replicated the experiments conducted in [2], adding the proposed method to the comparison. Second, we compare the performance of DOC, PROCLUS, LARFDSSOM/SS-SOM and ALTSS-SOM. Remark that the first two methods are originally from the data mining area. This choice of comparison methods was defined taking into consideration the analysis provided by [6], where LARFDSSOM presented the best results overall, and DOC and PROCLUS appeared as the two best options on average concerning subspace approaches in a distinction of data mining applications. Also, we refer the LARFDSSOM and SS-SOM together due to their equivalence for clustering tasks solely. Third, we performed a sensitivity analysis to show that with ALTSS-SOM the same range of parameters work well for a variety of datasets and that range does not exist for the previous methods (LARFDSSOM and SS-SOM).

For all experiments, the seven real-world datasets provided by the OpenSubspace framework [27] were used, rescaling them to the [0, 1] interval. Section V-A presents the experimental setup that was used. Next in the Section V-B, we present the results and analysis that are necessary to clarify the conclusions taken.

V-A Experimental Setup

For the first set of experiments, on every dataset, we used 3-times 3-fold cross-validation to measure the classification rate, as in [2]. Still as in [2], for studying the effects of the different the percentage of labeled data, the semi-supervised methods were trained with 1%, 5%, 10%, 25%, 50%, 75% and 100% of labels. In the second group of experiments, we have chosen the Clustering Error (CE) metric, as in [6] to evaluate the clustering assignments. For that, we considered all dimensions as relevant in the target clusterings used to calculate the CE. Also as in [6], we considered a problem of projected clustering, where each sample is assigned to a single cluster. Finally, we perform a sensitivity analysis with LARFDSSOM and ALTSS-SOM to elucidate the gains obtained regarding the importance that a parameter has in the outcome result. It will establish ALTSS-SOM as a more robust and self-controlled model.

For all the experiments, we sampled the parameter ranges according to a Latin Hypercube Sampling (LHS) [28], that guarantees the full coverage of the range of each parameter. In this sense, we gathered 500 different parameter settings, i.e., the range of each parameter was divided into 500 intervals of equal probability, resulting in a random selection of a single value from each interval [28]. The ranges used for ALTSS-SOM are defined in Table I, whereas the ranges of the other methods were the same as those used in [6, 2]. We set the maximum number of nodes for ALTSS-SOM to be 200.

V-B Experimental Results and Analysis

Fig. 1 shows the results of ALTSS-SOM in comparison with SS-SOM, Label Propagation and Label Spreading in the real-world datasets. The results are shown as a function of the percentage of labels that were used. Overall, the ALTSS-SOM improved the performance of SS-SOM, except for the Diabetes dataset (Fig. 1(b)) where the results obtained were slightly worse, but yet comparable. The flexibility provided by the estimation of the representing area allowed such results. Still, the standard deviation for all datasets in all supervision levels was also minimized, which indicates another positive aspect of the proposed method: it is more robust to variations on both datasets and parameters. With SS-SOM, the other semi-supervised methods surpassed its performance in the Pendigits and Vowel datasets, however, ALTSS-SOM achieved a consistent improvement by outperforming the results of both Label Propagation and Spreading in such cases. On all the other situations of this experiment, the ALTSS-SOM outperformed the comparing models.

Second, Table II shows the results of the CE obtained with the methods. It shows that no method achieved the best result for all real-world datasets. Even though the ALTSS-SOM presented an overall result that is better than all the others, it achieved the same results of DOC and LARFDSSOM for the Breast dataset. For the Glass dataset, again, it showed the same result of LARFDSSOM. Also, ALTSS-SOM was not the best for Diabetes, which shows a similar behavior when compared with the results obtained in the first set of experiments that took into account the classification rate. On considering a general comparison, ALTSS-SOM present the best results on average. It is worth mentioning that the similarity between the results of ALTSS-SOM and LARFDSSOM can be attributed to the fact that there were no labeled noise samples in the datasets, to the unknown information about the irrelevant dimensions, and to the intrinsic characteristics inherited by ALTSS-SOM from the LARFDSSOM. Also, once DOC does not have a direct way to control the number of clusters, it displays some difficulty to find out the correct value. Moreover, PROCLUS presented good results when the parameter controlling the number of clusters is defined close to the optimum. The good results obtained by LARFDSSOM is directly related to an excellent choice of the parameters $a_{t}$ and lp, which significantly impact the results. ALTSS-SOM achieves a good result without the needing of a highly accurate definition of parameters, as well as it is not necessary to define an a priori number of clusters due to its time-varying feature.

Third, Fig. 2 shows the scatter plot of the accuracy as a function of the parameter lp for the datasets trained with 50% of labels to illustrate a scenario where both forms of learning impact the outcome. Note that for all datasets, lp did not show a significant impact on the results. The linear fit to the data, represented by the red line, highlight this by not exhibiting any trend. It is also worth mentioning that the plots in Fig. 2 are the combination of each parameter value for each cross-validation set. In previous versions, the most two critical parameters were $a_{t}$ and lp. The parameter $a_{t}$ played a role of great importance due to its high impact on the results with just a small change on its values, i.e., $a_{t}$ impacted the results exponentially. Here in ALTSS-SOM, lp is the most important parameter because it defines more clearly the behavior of the algorithm. Despite it, it does not impact the result.

Fig. 3 also shows a scatter plot of $e_{b}$ parameter. However, we pick it up the datasets Breast and Pendigits to illustrate the choice of parameters. We first started with a wide range from 0.001 to 0.4 (Fig. 3(a) and Fig. 3(b)). However, the linear fit was mostly horizontal, not indicating any trend, again. We then shrank the range to 0.001 to 0.2 and reran the experiments. The results were as expected, keeping as stable as shown by Fig. 3(c) and Fig. 3(d). These experiments clarify how robust the model is to parameter changes, that is why we kept the same scale in the graphics. The two parameters taken for study in these figures were chosen due to its semantical importance, since other parameters presented similar behavior, with none of them acting a role as $a_{t}$ and lp in the before-mentioned versions.

VI Conclusion and Future Work

This paper presented our second approach for semi-supervised self-organizing maps applied to cluster and classification tasks. The behavior of ALTSS-SOM shown to have led to improvements on its previous version, the SS-SOM not only in terms of classification rate but also in clustering aspects. It consolidates the position of a good choice in situations where only a small portion of labels are available. The clustering task also achieved a significant improvement with the proposed changes. The ALTSS-SOM was able to reduce in two the number of parameters, whereby improving the performance in both contexts of classification and clustering metrics.

Also, probably one of the most important contributions of this current paper is related to the parametric robustness showed by the third and final experiment. It is of great importance to reduce the dependency and the variability of models to some parameters, and ALTSS-SOM achieves this.

The use of a relaxed estimated variance and region allows the method to better explore the information available on the data, improving sample efficiency. Also, the modifications proposed in ALTSS-SOM provided the ability to not merely discard data in certain cases but keep digging into its characteristics in order to establish a better estimate of their statistics.

Finally, we have left for future work some promising approaches that are related to defining a better stop criterion, to the use of the unsupervised error to build a model with more than one layer, as well as a hierarchical approach to provide better exploitation of the data statistics.

ACKNOWLEDGMENTS

The authors would like to thank CNPq (Conselho Nacional de Desenvolvimento Científico e Tecnológico), Brazil, for supporting this research study, and FACEPE (Fundação de Amparo à Ciência e Tecnologia do Estado de Pernambuco), Brazil, for financial support on the project #APQ-0880-1.03/14.