Preference Neural Network

TL;DR

This paper introduces a Preference Neural Network (PNN) with a novel activation function to effectively handle indifference preferences and multi-label ranking, demonstrating superior accuracy and efficiency on a new dataset.

Contribution

The paper presents a new neural network architecture with a smooth stairstep activation function designed for preference ranking with indifference and subgroup considerations.

Findings

PNN outperforms five existing methods in accuracy.

PNN achieves high computational efficiency.

PNN effectively handles indifference preferences in ranking.

Abstract

This paper proposes a preference neural network (PNN) to address the problem of indifference preferences orders with new activation function. PNN also solves the Multi-label ranking problem, where labels may have indifference preference orders or subgroups are equally ranked. PNN follows a multi-layer feedforward architecture with fully connected neurons. Each neuron contains a novel smooth stairstep activation function based on the number of preference orders. PNN inputs represent data features and output neurons represent label indexes. The proposed PNN is evaluated using new preference mining dataset that contains repeated label values which have not experimented before. PNN outperforms five previously proposed methods for strict label ranking in terms of accurate results with high computational efficiency.

Tables

Table 1. TABLE I: ANN types used in initial experiment.

Type	Ranker ANN	PNN
Activation Fun.	ReLU,Sigmoid	PSS, SS
Gradient	Descent	Ascent
Objective Fun.	RMS	$\rho$
Stopping Criteria.	$\tau$	$\rho$

Table 2. TABLE II: Benchmark data sets for label ranking; preference mining  [ 57 ] , real-world data sets  [ 58 ] and semi-synthetic ( s-s )  [ 52 ] .

Type	DS	Cat.	#Inst.	#Attr.	#lbl.
Mining	algae	chemical stat.	317	11	7
	german.2005	user pref.	413	31	5
	german.2009	user pref.	413	31	5
	sushi	user pref.	5000	13	7
	top7movies	user pref.	602	7	7
Real	cold	biology	2,465	24	4
	diau	biology	2,465	24	7
	dtt	biology	2,465	24	4
	heat	biology	2,465	24	6
	spo	biology	2,465	24	11
Semi-Synthesized	authorship	A	841	70	4
	bodyfat	B	252	7	7
	calhousing	B	20,640	4	4
	cpu-small	B	8192	6	5
	elevators	B	16,599	9	9
	fried	B	40,769	9	5
	glass	A	214	9	6
	housing	B	506	6	6
	iris	A	150	4	3
	pendigits	A	10,992	16	10
	segment	A	2310	18	7
	stock	B	950	5	5
	vehicle	A	846	18	4
	vowel	A	528	10	11
	wine	A	178	13	3
	wisconsin	B	194	16	16

Table 3. TABLE III: Comparison of classification on CIFAR-100  [ 54 ] and Fashion-Mnist data set  [ 55 ] using different convolution models

DS	Model	Baseline	MixUp
CIFAR-100	ResNet [59]	72.22	78.9
	WRN [60]	78.26	82.5
	Dense [61]	81.73	83.23
	EfficientNetV2-M [62]	92.2	-
	EffNet-L2 (SAM) [63]	96.08	-
	CvT [64]	94.39	-
	PrefNet	80.6	-
Fashion-MNIST	MLP	0.871	-
	RandomForest	0.873	-
	LogisticRegression	0.842	-
	SVC	0.897	-
	SGDClassifier	0.81	-
	LSTM [65]	0.8757	-
	DART [66]	0.965	-
	PrefNet	0.91	-

Table 4. TABLE IV: PNN performance comparison with various approaches: supervised clustering  [ 58 ] , supervised decision tree  [ 52 ] , MLP label ranking  [ 23 ] and label ranking tree forest ( LRT )  [ 67 ]

Label Ranking Methods
DS	S.Clust.	DT	MLP-LR	LRT	PNN
authorship	0.854	0.936(IBLR)	0.889(LA)	0.882	0.918
bodyfat	0.09	0.281(CC)	0.075(CA)	0.117	0.5591
calhousing	0.28	0.351(IBLR)	0.130(SSGA)	0.324	0.34
cpu-small	0.274	0.50(IBLR)	0.357(CA)	0.447	0.46
elevators	0.332	0.768(CC)	0.687(LA)	0.760	0.73
fried	0.176	0.99(CC)	0.660(CA)	0.890	0.91
glass	0.766	0.883(LRT)	0.818(LA)	0.883	0.8175
housing	0.246	0.797(LRT)	0.574(CA)	0.797	0.712
iris	0.814	0.966(IBLR)	0.911(LA)	0.947	0.917
pendigits	0.422	0.944(IBLR)	0.752(CA)	0.935	0.86
segment	0.572	0.959(IBLR)	0.842(CA)	0.949	0.916
stock	0.566	0.927(IBLR)	0.745(CA)	0.895	0.834
vehicle	0.738	0.862(IBLR)	0.801(LA)	0.827	0.754
vowel	0.49	0.90(IBLR)	0.545(CA)	0.794	0.85
wine	0.898	0.949(IBLR)	0.931(LA)	0.882	0.90
wisconsin	0.09	0.629(CC)	0.235(CA)	0.343	0.612
Average	0.475	0.79	0.621	0.730	0.755

Table 5. TABLE V: Comparison between PNN and supervised clustered on biological real world data in terms of L ​ o ​ s ​ s L ​ R 𝐿 𝑜 𝑠 subscript 𝑠 𝐿 𝑅 Loss_{LR}

Biological real world data
DS	S.Clustering	PNN
cold	0.198	0.11
diau	0.304	0.255
dtt	0.124	0.01
heat	0.072	0.013
spo	0.118	0.014
Average	0.1632	0.0804

Table 6. TABLE VI: PNN label ranking performance in terms of τ 𝜏 \tau coefficient, learning step and the number of middle layer neurons (#m.n). The training per fold and testing time is given in the last two columns. ‘s’, ‘m’, ‘h’ denote seconds, minutes and hours, respectively.

Type	DS	Avg. $\tau$	#m.n.	l.r.	#Iterations.	Dropout	Scaling.	Training t.	Testing t.
Real	cold	0.4	10	0.0008	2000	yes	-4:4	2.8h	1.2s
	diau	0.466	400	0.0005	2500	yes	-2:2	2.9h	4s
	dtt	0.60	400	0.0001	5000	yes	-4:4	5.7h	1.88s
	heat	0.876	450	0.0005	5000	yes	-2:2	6.2h	1.18s
	spo	0.8	300	0.0005	5000	yes	-4:4	7.4h	0.98s
	German2005	0.8	300	0.0005	1000	no	-4:4	35.15m	0.0879s
	German2009	0.67	300	0.0005	500	no	-4:4	7.087m	0.105s
Semi-Synthesized	authorship	0.931	200	0.0008	200	no	-4:4	3.82m	0.34s
	bodyfat	0.559	100	0.0005	2500	yes	-2:2	16.92m	0.44s
	calhousing	0.34	200	0.0007	1000	no	-2:2	5.03h	4.127s
	cpu-small	0.46	200	0.005	1000	no	-2:2	2.089h	1.717
	elevators	0.73	20	0.003	100	no	-2:2	27.03m	3.7s
	fried	0.89	100	0.005	100	no	-2:2	1.02h	8.45s
	glass	0.948	100	0.005	100	no	-3:3	14.8s	0.04s
	housing	0.7615	25	0.005	100	no	-3:3	37.21s	0.1s
	iris	0.956	100	0.005	100	no	-3:3	29.39s	0.066s
	pendigits	0.86	100	0.005	100	no	-3:3	34.6m	5.69s
	segment	0.956	20	0.007	100	no	-3:3	440.8s	0.94s
	stock	0.868	100	0.005	100	no	-3:3	142.48s	0.87s
	vehicle	0.869	100	0.005	100	no	-3:3	91s	0.2s
	vowel	0.85	100	0.005	100	no	-3:3	88.37s	0.312s
	wine	0.90	100	0.005	100	no	-3:3	19.19s	0.063s
	wisconsin	0.61	300	0.0005	2500	yes	-4:4	13.56m	0.1332s

Equations

$$\begin{split}y=-\frac{1}{2s}\Bigg{(}\sum_{i=0}^{n-1}\tanh&\left(c(wi-x)\right)-n\Bigg{)}\end{split}$$

$$\begin{array}{r}n=Y_{max}s\end{array}$$

$$\begin{array}{r}2b=w(n-1)\end{array}$$

$$\begin{split}y=-\frac{1}{2s}\Bigg{(}\sum_{i=0}^{n-1}\tanh&\left(c(b-x-wi)\right)-n\Bigg{)}\end{split}$$

$$\rho =1-\frac{6{\sum }_{i=1}^m(y_i-y{t}_i{)}^2}{m(m^2-1)}$$

$$dKw=2\cdot Im{g}_w-d\rho \cdot \frac{n^3-n}{-6}$$

$$\mathcal{O}(m\cdot o\cdot t)$$

$$W_m=W_m-Delt{a}_m$$

$$\mathcal{O}(p\cdot m\cdot o\cdot t)$$

$$\#Input=(Im{g}_w-K_w+1)\cdot (Im{g}_h-K_h+1)$$

$$\tau =1-2\cdot Los{s}_{LR}$$

Full text

††publicationid: pubid: 978-1-5386-5541-2/18/$31.00 ©2018 IEEE

I Introduction

PREFERENCE learning (PL) is an extended paradigm in machine learning that induces predictive preference models from experimental data [1, 2, 3]. PL has applications in various research areas such as knowledge discovery and recommender systems [4]. Objects, instances, and label ranking are the three main categories of PL domain. Of those, label ranking (LR) is a challenging problem that has gained importance in information retrieval by search engines [5, 6]. Unlike the common problems of regression and classification [7, 8, 9, 10, 11, 12, 13], label ranking involves predicting the relationship between multiple label orders. For a given instance x from the instance space $\mathbbm{x}$, there is a label $\pazocal{L}$ associated with x, $\pazocal{L}\in\pi$, where $\pi=\{\lambda_{1},..,\lambda_{n}$}, and n is the number of labels. LR is an extension of multi-class and multi-label classification, where each instance x is assigned an ordering of all the class labels in the set $\pazocal{L}$. This ordering gives the ranking of the labels for the given x object. This ordering can be represented by a permutation set $\pi=\{1,2,\cdots,n\}$. The label order has the following three features. irreflexive where $\lambda_{a}\nsucc\lambda_{a}$ ,transitive where ($\lambda_{a}\succ\lambda_{b})\land(\lambda_{b}\succ\lambda_{c}$) $\implies\lambda_{a}\succ\lambda_{c}$ and asymmetric $\lambda_{a}\succ\lambda_{b}\implies\lambda_{b}\nsucc\lambda_{a}$. Label preference takes one of two forms, strict and non-strict order. The strict label order ($\lambda_{a}\succ\lambda_{b}\succ\lambda_{c}\succ\lambda_{d}$) can be represented as $\pi=(1,2,3,4)$ and for non-restricted total order $\pi=(\lambda_{a}\succ\lambda_{b}\simeq\lambda_{c}\succ\lambda_{d})$ can be represented as $\pi=(1,2,2,3)$, where $a,b,c,and,d$ are the label indexes and $\lambda_{a},\lambda_{b},\lambda_{c}$ and $\lambda_{d}$ are the ranking values of these labels.

For the non-continuous permutation space, The order is represented by the relations mentioned earlier and the $\bot$ incomparability binary relation. For example the partial order $\lambda_{a}\succ\lambda_{b}\succ\lambda_{d}$ can be represented as $\pi=(1,2,0,3)$ where 0 represents an incomparable relation since $\lambda_{c}$ is not comparable to ($\lambda_{a},\lambda_{b},\lambda_{d}$).

Various label ranking methods have been introduced in recent years [14], such as decomposition-based methods, statistical methods, similarity, and ensemble-based methods. Decomposition methods include pairwise comparison [15, 16], log-linear models and constraint classification [17]. The pairwise approach introduced by Hüllermeier [18] divides the label ranking problem into several binary classification problems to predict the pairs of labels $\lambda_{i}\succ\lambda_{j}$ or $\lambda_{j}\prec\lambda_{i}$ for an input x. Statistical methods includes decision trees [19], instance-based methods (Plackett-Luce) [20] and Gaussian mixture model based approaches. For example, Mihajlo uses Gaussian mixture models to learn soft pairwise label preferences [21].

The artificial neural network (ANN) for ranking was first introduced as (RankNet) by Burge to solve the problem of object ranking for sorting web documents by a search engine [22]. Rank net uses gradient descent and probabilistic ranking cost function for each object pair. The multilayer perceptron for label ranking (MLP-LR) [23] employs a network architecture using a sigmoid activation function to calculate the error between the actual and expected values of the output labels. However, It uses a local approach to minimize the individual error per output neuron by subtracting the actual-predicted value and using Kendall error as a global approach. Neither direction uses a ranking objective function in backpropagation (BP) or learning steps.

The deep neural network (DNN) is introduced for object ranking to solve document retrieval problems. RankNet [22], RankBoost [24], and Lambda MART [25], and deep pairwise label ranking models  [26], are convolution neural Network (CNN) approaches for the vector representation of the query and document-based. CNN is used for image retrieval [27] and label classification for remote sensing and medical diagnosing [28, 29, 30, 31, 32, 33, 34, 35]. A multi-valued activation function has been proposed by Moraga and Heider [36] to propose a Generalized Multiple–valued Neuron with a differentiable soft staircase activation function, which is represented by a sum of a set of sigmoidal functions. In addition, Aizenberg proposed a generalized multiple-valued neuron using a convex shape to support complex numbers neural network and multi-values numbers [37]. Visual saliency detection using the Markov chain model is one approach that simulates the human visual system by highlighting the most important area in an image and calculating superpixels as absorbing nodes [38, 39, 40]. However, this approach needs a saliency optimization on the results and has calculation cost [41],[42].

Particle Swarm Optimization in movement detection is based on the concept of variation and inter-frame difference for feature selection. The swarm algorithms are mainly used in human motion detection in sports, and it is used based on probabilistic optimization algorithm [43, 44, 45, 46] and CNN [47].

Some of the methods mentioned above and their variants have some issues that can be broadly categorized into three types:

The ANN Predictive probability can be enhanced by limiting the output ranking values in the SS functions to a discrete value instead of a range of values of the rectified linear unit (Relu), Sigmoid, or Softmax activation functions. The predictive is enhanced by using the SS function slope as a step function to create discrete values, accelerating the learning by reducing the output values to accelerate the ranking convergence.

• 2)

The drawback of ranking based on the classification technique ignores the relation between multiple labels: When the ranking model is constructed using binary classification models, these methods cannot consider the relationship between labels because the activation functions do not provide deterministic multiple values. Such ranking based on minimizing pairwise classification errors differs from maximizing the label ranking’s performance considering all labels. This is because pairs have multiple models that may reduce ranking unification by increasing ranking pairs conflicts where there is no ground truth, which has no generalized model to rank all the labels simultaneously. For example, $\pazocal{D}=(1,1,1)$ for $\pi=(\lambda_{a}\succ\lambda_{b}\succ\lambda_{c})$ and $\pazocal{D}=(1,1,1)$ for $\pi=(\lambda_{a}\succ\lambda_{c}\succ\lambda_{b})$ the ranking is unique; however, pairwise classification creates no ground truth ranking for the pair $\lambda_{b}\succ\lambda_{c}$ and $\lambda_{c}\succ\lambda_{b}$ which adds more complexity to the learning process.

• 3)

Ignoring the relation between features. The convolution kernel has a fixed size that detects one feature per kernel. Thus, it ignores the relationship between different parts of the image. For example, CNN detects the face by combining features (the mouth, two eyes, the face oval, and a nose) with a high probability of classifying the subject without learning the relationship between these features. For example, the proposed PN kernel start attention to the important features that have a high number of pixel ranking variation.

The main contribution of the proposed neural network is

• •

Solving the label ranking as a machine learning problem.

• •

Solving the deep learning classification problem by employing computational ranking in feature selection and learning.

Where PNN has several advantages over existing label ranking methods and CNN classification approaches.

PNN uses the smooth staircase SS as an activation function that enhances the predictive probability over the sigmoid and Softmax due to the step shape that enhances the predictive probability from a range from -1 to 1 in the sigmoid to almost discrete multi-values.

• 2)

PNN uses gradient ascent to maximize the spearman ranking correlation coefficient. In contrast, other classification-based methods such as MLP-LR use the absolute difference of root mean square error (RMS) by calculating the differences between actual and predicted ranking and other RMS optimization, which may not give the best ranking results.

• 3)

PNN is implemented directly as a label ranker. It uses staircase activation functions to rank all the labels together in one model. The SS or PSS functions provide multiple output values during the conversions; however, MLP-LR and RankNet use sigmoid and Relu activation functions. These activation functions have a binary output. Thus, it ranks all the labels together in one model instead of pairwise ranking by classification.

• 4)

PN uses a novel approach for learning the feature selection by ranking the pixels and using different sizes of weighted kernels to scan the image and generate the features map.

The next section explains the Ranker network experiment, problem formulation, and the PNN components (Activation functions, Objective function, and network structure) that solve the Ranker problems and comparison between Ranker network and PNN.

II PNN Components

II-A Initial Ranker

The proposed PNN is based on an initial experiment to implement a computationally efficient label ranker network based on the Kendall $\tau$ error function and sigmoid activation function using simple structure as illustrated in section \Romannum4 Fig. 6.

The ranker network is a fully connected, three-layer net. The input represents one instance of data with three inputs, and there are six neurons in the hidden layer and three output neurons representing the labels’ index. Each neuron represents the ranking value. A small toy data set is used in this experiment. The ranker uses RMS gradient descent as an error function to measure the difference between the predicted and actual ranking values. The ranker has Kendall $\tau$ as a stopping criterion. The same ANN structure, number of neurons and learning rate using SS activation function, and spearman error function and gradient ascent of $\rho$ will be discussed in section \Romannum4. The ranking convergence reaches $\tau\simeq 1$ after 160 epochs using the Sigmoid function [48]. The sigmoid and ReLU shapes have a slightly high rate of change of $y$, and it produces a larger output range of data. Therefore, we consider ranking performance as one of the disadvantages of sigmoid function in the ranker network.

The ranker network has two main problems.

The ranker uses two different error functions, RMS for learning and Kendall $\tau$ for stopping criteria. Kendall $\tau$ is not used for learning because it is not continuous or differentiable. Both functions are not consistent as stopping criteria measure the relative ranking, and RMS does not, which may lead to incorrect stopping criteria. Enhancing the RMS may not also increase the error performance, as illustrated in Fig. 3 in a comparison between the ranker network. evaluation using $\rho$ and RMS.

• 2)

The convergence performance takes many iterations to reach the ranking $\tau\simeq 1$ based on the shape of sigmoid or Relu functions and learning rate as shown in the experiment video link [48] due to the slope shape between -1 or 0 and 1. The prediction probability almost equals the values from -1 or 0 to 1.

II-B Problem Formulation

For multi-class and multi-label problems, learning the data’s preference relation predicts the class classification and label ranking. i.e. data instance $\pazocal{D}\in\{x_{1},x_{2},\dots,x_{n}\}$. the output labels are predicted as ranked set labels that have preference relations $\pazocal{L}=\{{\lambda_{y}}_{1},\dots,{\lambda_{y}}_{n}\}$. PNN creates a model that learns from an input set of ranked data to predict a set of new ranked data. The next section presents the initial experiment to rank labels using the usual network structure.

II-C Activation Functions

The usual ANN activation functions have a binary output or range of values based on a threshold. However, these functions do not produce multiple deterministic values on the $y$-axis. This paper proposes new functions to slow the differential rate around ranking values on the $y$-axis to solve ranking instability. The proposed functions are designed to be non-linear, monotonic, continuous, and differentiable using a polynomial of the tanh function. The step width maintains the stability of the ranking during the forward and backward processes. Moraga [36] introduced a similar multi-valued function. However, the proposed exponential derivative was not applied to an ANN implementation. Moraga exponential function is geometrically similar to the step function [49]. However, The newly proposed functions consist of tanh polynomial instead of exponential due to the difficulty in implementation. The new functions detect consecutive integer values, and the transition from low to high rank (or vice versa) is fast and does not interfere with threshold detection.

II-C1 Positive Smooth Staircase (PSS)

As a non-linear and monotonic activation function, a positive smooth staircase (PSS) is represented as a bounded smooth staircase function starting from x=0 to $\infty$. Thus, it is not geometrically symmetrical around the $y$-axis as shown in Fig. 1. PSS is a polynomial of multiple tanh functions and is therefore differentiable and continuous. The function squashes the output neurons values during the FF into finite multiple integer values. These values represent the preference values from {0 to n} where 0 represents the incomparable relation $\bot$ and values from 1 to n represent the label ranking. The activation function is given in Eq. 1. PSS is scaled by increasing the step width $w$

[TABLE]

Where $n$ is the number of stair steps equal to the number of labels to rank, w is the step width, and $c$ is the stair curvature $c=100$ and $5$ for the sharp and smooth step, respectively. and $s$ is the scaling factor for reducing the height of each step to range to rank value with decimal place for the regression problems. $s$=10 and $s$=100 for 1 and 2 decimal places, respectively, $s$ is calculated as in Eq. 2.

[TABLE]

and $w$ is the step width as shown in Eq. 3.

[TABLE]

II-C2 Smooth Staircase (SS)

The proposed (SS) represents a staircase similar to (PSS). However, SS has a variable boundary value used as a hyperparameter in the learning process. The derivative of the activation function is discussed in section \Romannum3 and the performance comparison between SS and PSS is mentioned in Section \Romannum5.

The activation function is given in Eq. 4.

[TABLE]

where $c$ is step curvature, $n=$ number of ranked labels, $b$ is the boundary value on the x-axis, and (SS) lies between $-b$ and $b$.

where $Y_{max}$ is the max. value to rank. i.e. $Y_{max}$=3 and values have one decimal place. $n$ =30 The (SS) function has the shape of smooth stair steps, where each step represents an integer number of label ranking on the $y$-axis from 0 to $\infty$ as shown in Fig. 1, The SS step is not flat, but it has a differential slope. The function boundary value on the $x$-axis is from -$b$ to $b$ Therefore, input values must be scaled from -$b$ to $b$. The step width is 1 when n$\simeq 2b$. The convergence rate is based on the step width. However, it may take less time to converge based on network hyper parameters. Fig. 2 (a) and (b). The SS is scaled by increasing the boundary value $b$

II-D Ranking Loss Function

Two main error functions have been used for label ranking; Kendall $\tau$ [50] and spearman $\rho$ [51]. However, the Kendall $\tau$ function lacks continuity and differentiability. Therefore, the spearman $\rho$ correlation coefficient is used to measure the ranking between output labels. spearman $\rho$ error derivative is used as a gradient ascent process for BP, and correlation is used as a ranking evaluation function for convergence stopping criteria. $\tau_{Avg}$ is the average $\tau$ per label divided by the number of instances $m$, as shown in line 8 of Algorithm 1. spearman $\rho$ measures the relative ranking correlation between actual and expected values instead of using the absolute difference of root means square error (RMS) because gradient descent of RMS may not reduce the ranking error. For example, $\pi_{1}=(1,2.1,2.2)$ and $\pi_{2}=(1,2.2,2.1)$, have a low RMS $=0.081$ but a low ranking correlation $\rho=0.5$ and $\tau=0.3$.

Fig  3 shows the comparison between the initial ranker network and PNN; the ranker network uses Kendall $\tau$ which has lower performance as a stopping criterion compared to PNN spearman because the stopping criteria are based on the RMS per iteration; however, PNN uses spearman for both ranking step and stopping criteria.

The spearman error function is represented by Eq.5

[TABLE]

where $y_{i}$, $yt_{i}$, $i$ and $m$ represent rank output value, expected rank value, label index and number of instances, respectively.

II-E PNN Structure

II-E1 One middle layer

The ANN has multiple hidden layers. However, we propose PNN with a single middle layer instead of multi-hidden layers because ranking performance is not enhanced by increasing the number of hidden layers due to fixed multi-valued neuron output, as shown in Fig. 4; Seven benchmark data sets  [52] was experimented using SS function using one, two, and three hidden layers with the following hyper parameters; learning rate (l.r.)=0.05, and each layer has neuron $i=100$ and $b=10$). We found that by increasing the number of hidden layers, the ranking performance decreases, and more iterations are required to reach $\rho\simeq 1$. The low performance because of the shape of SS produces multiple deterministic values, which decrease the arbitrarily complex decision regions and degrees of freedom per extra hidden layer.

II-E2 Preference Neuron

Preference Neuron are a multi-valued neurons uses a PSS or SS as an activation function. Each function has a single output; however, PN output is graphically drawn by $n$ number of arrow links that represent the multi-deterministic values. The PN in the middle layer connects to only $n$ output neurons $stp=n+1$; where $stp$ is the number of SS steps. The PN in the output layer represents the preference value. The middle and output PNs produce a preference value from 0 to $\infty$ as illustrated in Fig. 5.

The PNN is fully connected to multiple-valued neurons and a single-hidden layer ANN. The input layer represents the number of features per data instance. The hidden neurons are equal to or greater than the number of output neurons, $H_{n}\geq\pazocal{L}_{n}$, to reach error convergence after a finite number of iterations. The output layer represents the label indexes as neurons, where the labels are displayed in a fixed order, as shown in Fig. 6.

\SetCoordinates

[xAngle =-35] \SetVertexStyle[MinSize = 8mm] \SetLayerDistance-3 \SetPlaneWidth5.5 \SetPlaneHeight5.5

The ANN is scaled up by increasing the hidden layers and neurons; however, increasing the hidden layers in PNN does not enhance the ranking correlation because it does not arbitrarily increase complex decision regions and degrees of freedom to solve more complex ranking problems. This limitation is due to the multi-semi discrete-valued activation function, limiting the output data variation. Therefore, instead of increasing the hidden layer, PNN is scaling up by increasing the number of neurons in the middle layer and scaling input data boundary value and increasing the PSS step width and SS boundaries which are equal to the input data scaling value, which leads to increased data separability.

PNN reaches ranking $\rho\simeq 1$ after 24 epochs compared to the initial ranker network that reaches the same result in 200 iterations, The video link demonstrates the ranking convergence as shown in Fig. 7 and video [48]. A summary of the three networks is presented in Table \Romannum1.

The output labels represent the ranking values. The differential PSS and SS functions to accelerate the convergence after a few iterations due to the staircase shape, which achieves stability in learning. PNN simplifies the calculation of FF and BP, and updates weights into two steps due to single middle layer architecture. Therefore, the batch weight updating technique is not used in PNN, and pattern update is used in one step. The network bias is low due to the limited preference neuron output of data variance; thus it is not calculated. Each neuron uses the SS or PS activation function in FF step, and calculates the preference number from 1 to n, where n is the number of label classes. During BP. The processes of FF and BP are executed in two steps until $\rho_{Avg}\simeq 1$ or the number of iterations reaches ($10^{6}$) as mentioned in the algorithm section.

The SS step width decreases by increasing the number of labels; thus, we increase function boundary $b$ to increase the step width to $\simeq 1$ to make the ranking convergence; In addition, a few complex data sets may need more data separability to enhance the ranking. Therefore, we use the $b$ value as a hyperparameter to keep the stair width $>=1$ and normalize input data from $-b$ to $b$.

Table \Romannum1 shows a brief comparison between Ranker ANN and PNN.

The following section describes the data preprocessing steps, feature selections, and components of PN.

III PN Components

III-A Image Preprocessing

III-A1 Greyscale Conversion

Data scaling as red, green, and blue (RGB) colors is not considered for ranking because PN measures the preference values between pixels. Thus, The image is converted from RGB color to Greyscale.

III-A2 Pixels’ Sorting

Ranking the image from $\pi=\{\lambda_{1},..,\lambda_{m}$} to $\pi=\{\lambda_{1},..,\lambda_{k}$} where the maximum greyscale value $\lambda_{m}=255$ and $\lambda_{k}$ is the maximum ranked pixel value as illustrated in Fig. 8 (a).

III-A3 Pixels Averaging

Ranking image pixels has an almost low ranking correlation due to noise, scaling, light, and object movement; therefore, window averaging is proposed by calculating the mean of pixel values of the small flattened window size of 2x2 of 4 pixels as shown in Fig. 9. The overall image $\rho$ of pixels increased from 0.2 to 0.79 in (a and b), from 0.137 to 0.75 for noisy images in (s and d), and scaled images from -0.18 to 0.71 in (e and f).

The two approaches, Pixel ranking and Averaging has been tested in remote sensing and faces images to detect the similarity, and it shows high ranking correlations using different window size as shown in Fig 10. It detects the high correlation by starting from the large window size $=$ image size. It reduces the size and scans until it reaches the highest correlation.

III-B Feature Selection By Attention

Feature selection for the kernel proceeded by selecting the features with a high group of pixel ranking variations indicating the importance of the scanned kernel area. This kind of hard attention makes the selection based on the threshold of pixel ranking values. to reduce the dimension of the input image.

III-C Feature Extraction

This paper proposes a new approach for image feature selection based on the preference values between pixels instead of the convolution of pixels array as implemented in CNN. The PN’s features are based on ranking computational space. Therefore, the kernel window size is considered a factor for feature selection.

III-C1 Pixels Resorting

The flattened window’s values are sorted for each kernel window in the image. The Fig. 8 (b) shows the window size 3X3 range from $\lambda_{k_{1}}=23$ to $\lambda_{k_{2}}=9$. Pixel sorting reduces the data margin, Thus, it reduces the computational complexity.

III-C2 Weighted Ranker Kernel

The kernel weights are randomly initialized from -0.05 to 0.05. The kernel learns the features by BP of its weights to select the best feature. the partial change in the kernel is calculated by differentiating the spearman correlation as in Eq. 6

[TABLE]

Different kernel sizes could be used for big images’ size. We use three different kernels to capture the relations between different features.

III-C3 Max Pooling

Max. pooling is used to reduce the features map’s size and select the highest correlation values to feed to the PNN.

\SetCoordinates

[xAngle =-35] \SetVertexStyle[MinSize = 8mm] \SetLayerDistance-3 \SetPlaneWidth5.5 \SetPlaneHeight5.5

III-D PN Structure

PN is the deep learning structure of PNN for image classification. It consists of five layers, a ranking features map and a max. pooling and three PNN layers. PN has one or multiple different sizes of PNNs connected by one output layer. Each PNN has SS or PSS where $\varphi_{n=2}$ for binary ranking to map the classification. The number of output neurons is the number of classes. The structure is shown in Fig 11. PN have one or more ranker kernels with different sizes, Each kernel has one corresponding PNN. PN uses the weighted kernel ranking to scan the image and extract the features map of spearman correlation values of the kernel with the scanned ranked image window as $\rho(\pi_{k},\pi_{w})$ where $\pi_{k}$ is the kernel preference values and $\pi_{w}$ is the scanned window image preference values. Each kernel scans the image by one step and creates a spearman features list. Max. Pooling is used to minimize the feature map used as input to PNN.

One 5X5 kernel is used for fashion Mnist data set [53]. Three kernels with sizes (3, 10, and 20) are used for CFAR-100 [54].

III-E Choosing The Kernel Size

Kernel size is chosen based on the hard attention of the highest group of pixels that has high ranking variation. The process scans the image sequentially starting from a small size to find the size with the highest pixels ranking variation. For example for the Mnist dataset where the image has a size of 28X28, The meaningful features are extracted using kernel sizes 10x10, 15x15, 20x20 and 25x25.

IV Algorithms

IV-A Baseline Algorithm

Algorithm 1 represents the three functions of the network learning process; feed-forward (FF), BP, and updating weights (UW). Algorithm 2 represents the learning flow of PN. Algorithm 3 represents the simplified BP function in two steps.

IV-B Ranking Visualization

PNN ranking convergence is visualized using the SS function by displaying the normalized input data points with corresponding actual ranked five labels represented in 5 different colours, The plotting of input value and SS output values per iteration is shown in Fig. 12, which illustrates the distribution of SS output values against the actual colour values at iterations 0 and 3900 and $\tau$ is enhanced from 0.39 to 0.85.

IV-C Complexity Analysis

IV-C1 Time Complexity

• •

FF time complexity corresponds to FF of middle and output layers, and $m$ and $n$ are the number of nodes in the middle and output layers. $W_{m}$ and $W_{o}$ are weighted matrix and $SS_{t}$ is the activation function of number of instances $t$. The time complexity in Eq. 7

[TABLE]

• •

BB starts with calculating the error of output layer $E_{ot}=\rho_{o}^{\prime}$ $Delta_{o}=E_{ot}\cdot SS^{\prime}$ and $Delta_{m}=E_{mt}\cdot SS^{\prime}$ then UW

[TABLE]

This time complexity is then multiplied by the number of epochs $p$

[TABLE]

IV-C2 Input Neurons

The number of PN input neurons is represented by Eq. 10

[TABLE]

where $w$ and $h$ are width and height of kernel and image.

V Network Evaluation

This section evaluates the PNN against different activation functions and architectures. All weights are initialized = 0 to compare activation functions and A and B have the same initialized random weights to evaluate the structure.

V-A Activation Functions Evaluation

PNN is tested on iris and stock data sets using four activation functions. SS, PSS, ReLU, sigmoid, and tanh. PNN has one middle layer and the number of hidden neurons (h.n.) is 50, while l.r.= 0.05. Fig. 13 shows the convergence after 500 iterations using four activation functions (SS, PSS, sigmoid, ReLU and tanh) respectively. We noticed that PSS and SS have a stable rate of ranking convergence compared to sigmoid, tanh, and ReLU. This stability is due to the stairstep width, which leads each point to reach the correct ranking during FF and BP in fewer epochs.

V-A1 PSS and SS Evaluation

As shown in Fig 13, PSS reaches convergence and remains stable for a long number of iterations compared to SS. However, SS has better $\rho$ than PSS. This good performance of SS is due to the reason:

• •

The symmetry of SS function on the $x$ axis. The SS shape handles both positive and negative normalized data. It reduces the number of iterations to reach the correct ranking values.

To have the same performance for SS and PSS, the input data should be scaled from 0 to step width X #steps and from -$b$ to $b$ for PSS and SS respectively.

V-A2 Missing Labels Evaluation

Activation functions are evaluated by removing a random number of labels per instance. PNN marked the missing label as -1; PNN neglects error calculation during BP, $\delta=0$. Thus, the missing label weights remain constants per learning iteration. The missing label approach is applied to the data set by 20% and 60% of the training data. The ranking performance decreases when the number of missing labels increases. However, SS and PSS have more stable convergence than other functions. This evaluation is performed on the iris data set, as shown in Fig. 13.

V-A3 Statistical Test

The PNN results were evaluated using receiver operating characteristic (ROC) curves. The true positive and negative for each rank are evaluated per label of wine dataset as shown in Fig. 14. The confusion matrix on wine and glass DS are shown in Fig. 15 where $\tau$ = 0.947, 0.84, Accuracy = 0.935 and 0.8 in (a) and (b) respectively.

V-A4 Dropout Regularization

Dropout is applied as a regularization approach to enhance the PNN ranking stability by reducing over-fitting. We drop out the weights that have a probability of less than 0.5. these dropped weights are removed from FF, BP, and UW steps. The comparison between dropout and non-dropout of PNN are shown in Fig. 16. The gap between the training model and ten-fold cross-validation curves has been reduced using dropout regularization using hyperparameters (l.r.=0.05, h.n.=100) on the iris data set. The dropout technique is used with all the data ranking results in the next section.

The following section is the evaluation of ranking experiments using label benchmark data sets.

VI Experiments

This section describes the classification and label ranking benchmark data sets, the results using PN and PNN, and a comparison with existing classification and ranking methods.

VI-A Data sets

VI-A1 Image Classification Data sets

PN is evaluated using CFAR-100 [54] and Fashion-MNIST [55] data sets.

VI-A2 Label Ranking Data sets

PNN is experimented with using three different types of benchmark data sets to evaluate the multi-label ranking performance. The first type of data set focuses on exception preference mining [56], and the ‘algae’ data set is the first type that highlights the indifference preferences problem, where labels have repeated preference value [57]. German elections 2005, 2009, and modified sushi are considered new and restricted preference data sets. The second type is real-world data related to biological science [18]. The third type of data set is semi-synthetic (SS) taken from the KEBI Data Repository at the Philipps University of Marburg [52]. All data sets do not have ranking ground truth, and all labels have a continuous permutation space of relations between labels. Table \Romannum2 summarizes the main characteristics of the data sets.

VI-B Results

VI-B1 Image Classification Results

PN has 3 kernel sizes of 5,10 and 20 and is tested on the CFAR-100 [54] data set and 1 kernel with a size 5 for Fashion-MNIST data set [55]. Table \Romannum3 shows the results compared to other convolutions networks.

VI-B2 Label Ranking Results

PNN is evaluated by restricted and non-restricted label ranking data sets. The results are derived using spearman $\rho$ and converted to Kendall $\tau$ coefficient for comparison with other approaches. For data validation, we used 10-fold cross-validation. To avoid the over-fitting problem, We used hyperparameters, i.e. l.r.= (0.0008,0.0005,0.005, 0.05, 0.1) hidden neuron = no.inputs+(5, 10, 50, 100, 200,300,400,450) neurons and scaling boundaries from 1 to 250) are chosen within each cross-validation fold by using the best l.r. on each fold and calculating the average $\tau$ of ten folds. Grid searching is used to obtain the best hyperparameter. For type B, we use three output groups and l.r.=0.001 and $w_{b}=0.01$.

VI-B3 Benchmark Results

Table \Romannum4 summarizes PNN ranking performance of 16 strict label ranking data sets by l.r. and m.n. The results are compared with the four methods for label ranking; supervised clustering [58], supervised decision tree [52], MLP label ranking [23], and label ranking tree forest (LRT) [67]. Each method’s results are generated by ten-fold cross-validation. The comparison selects only the best approach for each method.

During the experiment, it was found that ranking performance increases by increasing the number of central neurons up to a maximum of 20 times the number of features. As shown in Table \Romannum6, The real datasets are ranked using PNN with dropout regulation due to complexity and over-fitting. The dropout requires increasing the number of epochs to reach high accuracy. All the results are held using a single hidden layer with various hidden neurons (100 to 450) and SS activation function. The Kendall $\tau$ error converges and reaches close to 1 after 2000 iterations, as shown in Fig. 17.

Table \Romannum4 compares PNN with similar approaches used for label ranking. These approaches are; Decision trees [58], MLP-LR [23] and label ranking trees forest LRT [67]. In this comparison, we choose the method that has the best results for each approach.

VI-B4 Preference Mining Results

The ranking performance of the new preference mining data set is represented in table \Romannum2. Two hundred fifty hidden neurons are used To enhance the ranking performance of the algae data set’s repeated label values. However, restricted labels ranking data sets of the same type, i.e., (German elections and sushi), did not require a high number of hidden neurons and incurred less computational cost.

Experiments on the real-world biological data set were conducted using supervised clustering (SC) [58], Table \Romannum5 presents the comparison between PNN and supervised clustering on biological real world data in terms of $Loss_{LR}$ as given in Eq. 11.

[TABLE]

where $\tau$ is Kendall $\tau$ ranking error and $Loss_{LR}$ is the ranking loss function.

SS function with 16 steps is used to rank Wisconsin data set with 16 labels. By increasing the number of steps in the interval and scaling up the features between -100 and 100, The step width is small. To enhance ranking performance, the data set has many labels. The number of hidden neurons is increased to exceed $\tau=0.5$.

VI-C Computational Platform

PNN and PN is implemented from scratch without the Tensorflow API and developed using Numba API to speed the execution on the GPU and use Cuda 10.1 and Tensorflow-GPU 2.3 for GPU execution and executed at the University of Technology Sydney High-Performance Computing cluster based on Linux RedHat 7.7, which has an NVIDIA Quadro GV100 and memory of 32 G.B. For a non-GPU version of PNN is located at GitHub Repository [68].

VI-D Discussion and Future Work

It can be noticed from table \Romannum2 that PN is performing better than ResNet [59] and WRN [60]. Different types of architectures of PN could be used to enhance the results and reach state-of-the-art in terms of image classification [69, 70, 71]. It can be noticed from table \Romannum3 that PNN outperforms on SS data sets with $\tau_{Avg}=0.8$, whereas other methods such as, supervised clustering, decision tree, MLP-ranker and LRT, have results $\tau_{Avg}=0.79,0.73,0.62,0.475,$ respectively. Also, the performance of PNN is almost 50% better than supervised clustering in terms of ranking loss function $Loss_{LR}$ on real-world biological data set, as shown in table \Romannum5. The superiority of PNN is used for classification and ranking problems. The ranking is used in input data as a feature selection criteria is a novel approach for deep learning.

Encoding the labels’ preference relation to numeric values and ranking the output labels simultaneously in one model is an advanced step over pairwise label ranking based on classification. PNN could be used to solve new preference mining problems. One of these problems is incomparability between labels, where Label ranking has incomparable relation $\bot$, i.e., ranking space ($\lambda_{a}\succ\lambda_{b}\bot\lambda_{c}$) is encoded to (1, 2, -1) and $(\lambda_{a}\succ\lambda_{b})\bot(\lambda_{c}\succ\lambda_{d})$ is encoded to (1, 2, -1, -2). PNN could be used to solve new problem of non-strict partial orders ranking, i.e., ranking space ($\lambda_{a}\succ\lambda_{b}\succeq\lambda_{c}$) is encoded to (1, 2, 3) or (1, 2, 2). Future research may enhance PN by adding kernel size and SS parameters as part of the deep learning to choose the best kernel size and SS step width, which could enhance the image attention. Modifying PNN architecture by adding bias and solving noisy label ranking problems.

VII Conclusion

This paper proposed a novel method to rank a complete multi-label space in output labels and features extraction in both simple and deep learning.PN is a new research direction for image recognition based on new kernel and pixel calculations. PNN and PN are native ranker networks for image classification and label ranking problems that uses SS or PSS to rank the multi-label per instance. This neural network’s novelty is a new kernel mechanism, activation, and objective functions. This approach takes less computational time with a single middle layer. It is indexing multi-labels as output neurons with preference values. The neuron output structure can be mapped to integer ranking value; thus, PNN accelerates the ranking learning by assigning the rank value to more than one output layer to reinforce updating the random weights. PNN is implemented using python programming language 3.6 [68], and activation functions are modelled using wolframe Mathematica software [72]. A video demo that shows the ranking learning process using toy data is available to download [48].

Acknowledgement

This work was supported in part by the Australian Research Council (ARC) under discovery grants DP180100656 and DP210101093. The research was also sponsored in part by the US Office of Naval Research Global under Cooperative Agreement Number ONRG - NICOP - N62909-19-1-2058 and AFOSR – DST Australian Autonomy Initiative agreement ID10134. We also thank the NSW Defence Innovation Network and NSW State Government of Australia for financial support in part of this research through grant PP21-22.03.02.