Multi-Valued Neural Networks I A Multi-Valued Associative Memory

TL;DR

This paper introduces a novel multi-valued associative memory model using lattice theory, expanding fuzzy neural network concepts to handle elements or subsets of a lattice for classification tasks.

Contribution

It generalizes fuzzy associative memory to multi-valued cases without numerical representations, providing new properties and a learning algorithm for such networks.

Findings

Defined conditions for storing pattern pairs in the memory

Developed a learning algorithm for multi-valued networks

Applied the model to classify aircraft/spacecraft trajectories

Abstract

A new concept of a multi-valued associative memory is introduced, generalizing a similar one in fuzzy neural networks. We expand the results on fuzzy associative memory with thresholds, to the case of a multi-valued one: we introduce the novel concept of such a network without numbers, investigate its properties, and give a learning algorithm in the multi-valued case. We discovered conditions under which it is possible to store given pairs of network variable patterns in such a multi-valued associative memory. In the multi-valued neural network, all variables are not numbers, but elements or subsets of a lattice, i.e., they are all only partially-ordered. Lattice operations are used to build the network output by inputs. In this paper, the lattice is assumed to be Brouwer and determines the implication used, together with other lattice operations, to determine the neural network output. We gave the example of the network use to classify aircraft/spacecraft trajectories.

Equations

$$y_j=\underset{i}{\bigvee }\{x_i\wedge w_{ij}\}.$$

$$y_j=(\underset{i}{\bigvee }\{(x_i\vee c_{ij})\wedge w_{ij}\})\vee d_j=\underset{i}{\bigvee }\{(x_i\vee c_{ij}\vee d_j)\wedge (w_{ij}\vee d_j)\}.$$

$$\text{y}=((\text{x}\vee \text{c})\circ \text{W})\vee \text{d}.$$

$$\begin{gathered} G_{ij}(X,Y)=\{k\in P|x_i^k>y_j^k\},E_{ij}(X,Y)=\{k\in P|x_i^k=y_j^k\}, \\ G{E}_{ij}(X,Y)=G_{ij}(X,Y)\cup E_{ij}(X,Y),L_{ij}(X,Y)=\{k\in P|x_i^k<y_j^k\}, \\ L{E}_{ij}(X,Y)=L_{ij}(X,Y)\cup E_{ij}(X,Y), \\ N{C}_{ij}(X,Y)=\{k\in P|x_i^k\notin G{E}_{ij}(X,Y)\cup L{E}_{ij}(X,Y)\}. \end{gathered}$$

$$d_j^0=\underset{k\in P}{\bigwedge }(y_j^k),$$

$$c_{ij}^0=\{\begin{array}{r}\underset{\begin{array}{c}k\in L{E}_{ij}(X,Y),\\ j\in M\end{array}}{\bigwedge }(y_j^k),L{E}_{ij}(X,Y)\ne \varnothing ;\\ \underset{\begin{array}{c}k\in P,\\ j\in M\end{array}}{\bigwedge }(y_j^k),L{E}_{ij}(X,Y)=\varnothing ,\end{array}$$

$$w_{ij}^0=\underset{k\in P}{\bigwedge }(x_i^k\Rightarrow y_j^k).$$

$$\begin{gathered} T{G}_{ij}((X,{\text{c}}^0);{\text{d}}^0,Y)=\{k\in P|x_i^k\vee c_{ij}^0\vee d_j^0>y_j^k\}, \\ T{E}_{ij}((X,{\text{c}}^0);{\text{d}}^0,Y)=\{k\in P|x_i^k\vee c_{ij}^0\vee d_j^0=y_j^k\}, \\ T{L}_{ij}((X,{\text{c}}^0);{\text{d}}^0,Y)=\{k\in P|x_i^k\vee c_{ij}^0\vee d_j^0<y_j^k\}, \\ TG{E}_{ij}((X,{\text{c}}^0);{\text{d}}^0,Y)=T{G}_{ij}((X,{\text{c}}^0);{\text{d}}^0,Y)\cup T{E}_{ij}((X,{\text{c}}^0);{\text{d}}^0,Y)\}, \\ TL{E}_{ij}((X,{\text{c}}^0);{\text{d}}^0,Y)=T{L}_{ij}((X,{\text{c}}^0);{\text{d}}^0,Y)\cup T{E}_{ij}((X,{\text{c}}^0);{\text{d}}^0,Y)\}, \\ TN{C}_{ij}((X,{\text{c}}^0);{\text{d}}^0,Y)=\{k\in P|k\notin TG{E}_{ij}((X,{\text{c}}^0);{\text{d}}^0,Y)\cup \\ \cup TL{E}_{ij}((X,{\text{c}}^0);{\text{d}}^0,Y)\}, \\ T{S}_{ij}^G(({\text{W}}^0,\text{d});Y)=\{k\in P:\mid x_i^k\wedge y_j^k⩽w_{ij}^0\vee d_j^0\}. \end{gathered}$$

$$T{S}_{ij}^G(({\text{W}}^0,\text{d});Y)=\{k\in P:\mid x_i^k\wedge y_j^k⩽w_{ij}^0\}.$$

$$M^{wcd}=\{(\text{W},\text{c},\text{d})|\forall k\in P,(({\text{x}}^k\vee \text{c})\circ \text{W})\vee \text{d}={\text{y}}^k\}.$$

$$(x_i^k\vee c_{ij}\vee d_j)\wedge (w_{ij}\vee d_j)⩽(x_i^k\vee c_{ij}^0\vee d_j^0)\wedge (w_{ij}^0\vee d_j^0)⩽y_j^k.$$

$$\begin{gathered} y_j^k=\underset{i\in N}{\bigvee }\{(x_i^k\vee c_{ij}\vee d_j)\wedge (w_{ij}\vee d_j)\}⩽ \\ ⩽\underset{i\in N}{\bigvee }\{(x_i^k\vee c_{ij}^0\vee d_j^0)\wedge (w_{ij}^0\vee d_j^0)\}⩽y_j^k. \end{gathered}$$

$$w_{nj}^0⩾x_n^k\wedge y_j^k,$$

$$\underset{n:k\in T{S}_{nj}^G}{\bigvee }{x}_n^k\wedge y_j^k=y_j^k.$$

$$\begin{gathered} \underset{i\in N}{\bigvee }((x_i^k\vee c_j\vee d_j)\wedge (w_{ij}^0\vee d_j))⩾ \\ ⩾\underset{i\in N}{\bigvee }(x_i^k\wedge w_{ij}^0)\wedge y_j^k⩾ \\ ⩾\underset{n:k\in T{S}_{nj}^G}{\bigvee }(x_n^k\wedge w_{nj}^0\wedge y_j^k)=y_j^k, \end{gathered}$$

$$\underset{i\in N}{\bigvee }\{(x_i^k\vee c_j\vee d_j)\wedge (w_{ij}^0\vee d_j)\}\in \{x_i^l\in L\mid l\in G{E}_{ij}(X,Y)\}.$$

$$\underset{i\in N}{\bigvee }\{(x_i^k\vee c_j\vee d_j)\wedge (w_{ij}^0\vee d_j)\}\in \{x_i^l\in L\mid l\in L{E}_{ij}(X,Y)\}.$$

$$\{w{\}}_{ij}^k(t+1)=\{\begin{array}{r}\{w{\}}_{ij}^k(t);ifk:[(x_i^k\vee c_{ij})\wedge w_{ij}(t)]\vee d_j⩽y_j^k;\\ \{w{\}}_{ij}^k(t)-(\{x_i^k\Rightarrow y_j^k(t)\}-\{x_i^k\Rightarrow y_j^k\}),otherwise.\end{array}$$

$$\{w{\}}_{ij}(t+1)=\bigcap _k\{w{\}}_{ij}^k(t+1).$$

$$y_j^k=(\underset{i}{\bigvee }\{(x_i^k\vee c_{ij})\wedge w_{ij}\})\vee d_j=\underset{i}{\bigvee }[(x_i^k\wedge w_{ij})\vee (c_{ij}\wedge w_{ij})]\vee d_j.$$

$${{\text{W}}^0}^T=\left(\begin{array}{cccc}cd,& cb,& ac,& c\\ cd,& c,& bcd,& bc\end{array}\right).$$

Full text

††journal: Neural Computing and Applications. 2021.

Vol. 33 (16). P. 10189-10198.

1 Introduction

Feedforward fuzzy neural networks in which the internal operations based on fuzzy operations of joins and meets $\vee-\wedge$, were proposed in Kosko (1987). Such structures are called fuzzy associative memory, and fuzzy information in them is represented by the elements of the numeric interval [0, 1]. These networks are used for storing and classifying of fuzzy patterns. More complicated variants of fuzzy networks use fuzzy numbers as weights and variables. The most significant use of such networks lies in the field of approximation analysis and image restoration Liu and Li (2004). Also, there is a self-organizing fuzzy network which reorganize the model and adapt itself to a changing environment de Jesus Rubio (2009).

However, often, it is preferable to operate not with number representation of information, but with linguistic expressions directly. Such a situation occurs, e.g., in decision making systems Xu et al. (2010) and cognitive maps. For example, when fuzzy assessing typical flight situations, a set of linguistic variables and their degrees of significance/certainty, estimate the emerging state. Here, confidence levels take, in turn, values in a numerical interval, usually [0,1]. However, a problem arises here: the methods for determining fuzziness are subjective — experts assess the degree of fuzziness that implies some uncertainty.

In Maximov (2019a) it was demonstrated that it is not necessary to use numbers in such a situation: linguistic variables (but already partially-ordered, as it mentioned in Xu et al. (2010)), which do not require a mandatory numerical evaluation, can serve again as assessments. The situation itself determines such estimations that do not need an expert opinion or, at least, the expert’s evaluation of the situation is greatly facilitated.

In this case, to compare the valuations and obtain control solutions in Maximov (2019a), the concepts of not fuzzy logic are used, but multi-valued, in which the scale of truth values is a Brouwer lattice of a general form. Such scales of truth values generalize a linearly ordered scale (in particular, in fuzzy logic) Maximov (2016a) and naturally define implication as a lattice operation. Different variants of such multi-valued or lattice-valued logics111We use the term “multi-valued” for the special case of finite and non-linearly-ordered scale of truth values used in these logics. can be found in Novak (1982), Xu et al. (2003), Maximov (2016b), Maximov (2018).

Such decision making systems can use neural networks in their processing. However, nobody has used general non-numeric multi-valued lattices for neural networks weights and data representation so far. Though, lattice-valued neural networks are known Liu and Wang (1997) as fuzzy ones generalization, variables in these networks are represented by set-valued functions with the sets are again numeric subintervals of an interval in $\mathbb{R}$.

Thus, the main contribution of the paper is the first example of a general associative memory in which weights and data take values in a non-numeric multi-valued linguistic lattice and an application of such a network as a simple linguistic patterns classifier. However, the lattice does not have to be a linguistic variable one. Such a lattice may be a lattice of sets or, even, a lattice of graphs of system state configurations as in the researches on a system state estimating, not by numbers, but by elements of a lattice: Maximov et al. (2017), Maximov (2019b),

Maximov and Ryvkin (2017), Maximov and Ryvkin (2019).

We will use elements of a complete lattice in a multi-valued associative memory instead of elements of the interval [0, 1]. In this paper we suppose the lattice is Brouwer. The case of a residuated lattice is considered in Maximov (2020c). The associative memory generalizes a fuzzy one of

Liu and Li (2004) to the case of using multi-valued logic operations (instead of fuzzy ones) with the multi-valued lattice elements.

In Maximov (2020a), Maximov (2020b) the simplest case of a multi-valued associative memory without thresholds has been considered. In this paper, we expand these results (as well as Liu and Li (2004) on fuzzy associative memory with thresholds), to the case of a multi-valued associative memory with thresholds: we define the concept of such a network, investigate its properties, and give a generalization of the learning algorithm of fuzzy associative memory for a multi-valued case. Though, the inputs, outputs and connection weights of the network are linguistic variables, not numbers, this makes it possible to use such a network for processing control and diagnostic information of complex dynamic objects. We demonstrate this application by such a network use to classify aircraft/spacecraft trajectories.

The paper is organized as follows: in Sec. 2 we give a brief list of definitions used in the text. In Sec. 3 we define the multi-valued associative memory and prove some existence theorems for the solution of the equation defining the memory. In Sec. 4 we consider a learning algorithm for the memory. In Sec. 5 we discuss the previous results and the network computational complexity. In Sec. 6 we give an example of such a network use as a linguistic pattern classifier. In Sec. 7 we conclude the paper.

2 Definitions

Definition 1

A lattice is a partially-ordered set having, for any two elements, their exact upper bound or join $\vee$ (sup, max) and the exact lower bound or meet $\wedge$ (inf, min).

Definition 2

The exact upper bound of the two elements is the smallest lattice element, larger than both of these elements.

Definition 3

The exact lower bound is dually defined as the largest element of the lattice, smaller than both the elements.

Definition 4

A complete lattice is a lattice in which any two subsets have a join and a meet. It follows from the definition that in a non-empty complete lattice, there is the biggest “1” and the smallest “0” elements.

Definition 5

Generators of the lattice are called its elements, from which all the others are obtained by join and meet operations.

Definition 6

A lattice is called atomic if every two of its generators have null meets.

If we take such a lattice as a scale of truth values in a multi-valued logic, then the biggest element will correspond to complete truth (true), the smallest to complete falsehood (false), and intermediate elements will correspond to partial truth in the same way as the elements of the segment [0,1] evaluate partial truth in fuzzy logic.

In logics, with such a scale of truth values, the implication can be determined by multiplying lattice elements, or internally, only from lattice operations.

Definition 7

A Brouwer lattice is a lattice that has internal implications.

Definition 8

In such a lattice, the implication $c=a\Rightarrow b$ is defined as the largest $c:\;a\wedge b=a\wedge c$.

Definition 9

The implication $\neg a=a\Rightarrow 0$ is called the pseudo-complement of $a$.

Distribution laws for join and meet are satisfied in Brouwer lattices. The converse is valid only for finite lattices.

We will assume that the lattices used in the multi-valued neural network are complete, distributive and finite.

3 Multi-Valued $\vee-\wedge$ Associative Memory with Threshold

As in the case of fuzzy associative memory Liu and Li (2004), in $\vee-\wedge$ multi-valued associative memory, the transfer function normalizing outputs to the prefix range is not needed, since the $\vee-\wedge$ operators limit the outputs to the range of inputs, and $\wedge$ is also a threshold operator

Blanco et al. (1995). A $\delta$–function which activates the desired neuron in a one-layer multi-valued net computing a lattice implication, is considered in Maximov (2020b) (see also Sec 5).

More complicated fuzzy neural networks, which work not with elements of $[0,1]$, but with fuzzy numbers, and in need also of activation, use different transfer functions, from simplest to a uniform Tauber-Wiener function Liu and Li (2004). However, a theory of a multi-valued analogue of fuzzy numbers might be developed in order to build complicated multi-valued neural networks with transfer functions. The idea of SPOCU function “picking up” the appropriate properties for activation function directly from training Kiselak et al. (2020) may be useful here (see also, Maximov D. “Multi-Valued Cognitive Maps”).

Let us suppose, that an input signal is $x\in L^{n}$, and an output signal is $y\in L^{m}$, where $L$ is the lattice used. In this case, the input-output relationship in a two-layer multi-valued associative memory can be written as $\textbf{y}=\textbf{x}\circ\textbf{W}$, where $\circ$ stands for the $\vee-\wedge$ composition operation and $\textbf{W}=(w_{ij})_{n\times m}\in\mu_{n\times m}$ is the $n\times m$ matrix of the weights of connections, with elements from a complete, distributive, and finite lattice $L$, $i\in N=\{1...n\},\;j\in M=\{1...m\}$ (Fig. 1).

In Maximov (2020b), several simple composition options are suggested, one of them is:

[TABLE]

In the theory of the fuzzy associative memory Liu and Li (2004), exactly (1) is considered, however, with fuzzy values and operations. There, the fuzzy operator $\vee$ generalizes the sum $\sum$ in ordinary neural networks, and the fuzzy operator $\wedge$ is used as a special case of multiplication. In multi-valued neural networks we use in (1) the lattice operations $\vee$ and $\wedge$ with the same functionality, and $\wedge$ is again a special case of multiplication in residuated lattices. More sophisticated fuzzy models use a combination of $\vee$ and $t$-norms Liu and Li (2004), Sussner and Valle (2006) and a combination with implication Sussner and Valle (2006). However, because we use not a linear-ordered segment as a set of variables, but the general Brouwer lattice without multiplying elements (in this paper), we do not consider these models.

Thus, we consider the composition 1, but with thresholds $c_{ij}$ and $d_{j}$ to input unit $i$ and output unit $j$ respectively (Fig. 1) with all variables taking values in the lattice $L$:

[TABLE]

In the vector notation, it can be written as:

[TABLE]

Here, all quantities are the elements (not subsets!) of the lattice $L$. Hence, no membership functions are needed unlike the fuzzy case. As with the activation function, the theory of a multi-valued analogue of fuzzy numbers should be developed in order to consider membership functions. Then, such multi-valued numbers may be used also in, e.g., quality modelling of a non-linear process similar to Aquino and et al (2020) or membership function construction as in Chiang et al. (2019).

We denote $(X,Y)=\{\textbf{x}^{k},\textbf{y}^{k}\;|\;k\in P\},\;P=\{1...p\}$ — the family of pairs of multi-valued patterns with $\textbf{x}^{k}=(x_{1}^{k},...,x_{n}^{k})$ and $\textbf{y}^{k}=(y_{1}^{k},...,y_{m}^{k})$. We introduce also the following sets (Fig. 2):

[TABLE]

The equalities define those sets of pattern pairs, for which $x$’s are grater than $y$’s (G), or less than y’s (L), … or not comparable with $y$’s (NC).

We also establish the threshold matrix $\textbf{c}^{0}=(c^{0}_{11},...,c^{0}_{nm})$, and vector $\textbf{d}^{0}=(d^{0}_{1},...,d^{0}_{m})$, and the connection weight matrix $W_{0}=(w_{ij}^{0})_{n\times m}$ like in

Maximov (2020a) and in a different way from Liu and Li (2004):

[TABLE]

[TABLE]

[TABLE]

The input thresholds $c^{0}$’s are individual for each input-output connection. One should add an additional meet by $i$ in (6) to obtain the unique threshold for all inputs. This is the case of $LE_{ij}(X,Y)=\emptyset$ in (6). These constants $c^{0}$’s raise to the level of outputs’ intersections the inputs in patterns where $x$’s less than or equal to $y$’s. Thus, the capacity of the memory increases, since we can store more patterns with small $x$’s (Theorem 3). Also, $d^{0}$’s adapts everything to the memory outputs.

Now, we define the sets ($i\in N,\;j\in M$):

[TABLE]

The sets are similar to ones without “$T$” in denotation above (4), however these ones use join of $x$’s with $c^{0}$’s and $d^{0}$’s instead of only $x$’s. The set $TS$ determine patterns for which weights $w^{0}$’s are grater than or equal to outputs, since, $d_{j}^{0}\leqslant w_{ij}^{0}$ by (7), (5) and the implication Definition 8 ($c\geqslant b$ in $c=a\Rightarrow b$):

[TABLE]

Then, let us define the set

[TABLE]

This is the set of the pattern pair family, thresholds, and the connection matrix, which satisfy the equation (2). The following Theorem shows that weights $w$’s and thresholds $d$’s which give equation (2) solutions, are bounded from above by the values of these $w^{0}$’s and $d^{0}$’s.

Theorem 1

Let $\textbf{W}=(w_{ij})_{n\times m}\in\mu_{n\times m},\;\textbf{c}=(c_{11},...,c_{nm})\in L^{n\times m},\;\textbf{d}=(d_{1},...,d_{m})\in L^{m},\;(\textbf{W},\textbf{c},\textbf{d})\in M^{wcd}.$ Then $\forall i\in N,\;j\in M,\;w_{ij}\leqslant w_{ij}^{0},\;d_{j}\leqslant d_{j}^{0}$.

Proof

Since $(\textbf{W},\textbf{c},\textbf{d})\in M^{wcd}$, we denote in (3) $(\textbf{x}\vee\textbf{c})\circ\textbf{W}=\textbf{U}$ and get $\textbf{U}\vee\textbf{d}=\textbf{y}$. Taking the meet by $k$ of this equation, we get $\bigwedge_{k\in P}(U^{k})\vee d_{j}=\bigwedge_{k\in P}(y_{j}^{k})=d_{j}^{0}$, since $d_{j}$’s do not depend on $k$. Thus, $d_{j}\leqslant d_{j}^{0}$.

Hence next, we proof $w_{ij}\leqslant w_{ij}^{0}$. Clear, that if $b>c$, then $a\Rightarrow b\geqslant a\Rightarrow c$. Thus, $w_{ij}^{0}=\bigwedge_{k\in P}(x_{i}^{k}\Rightarrow y_{j}^{k})=\bigwedge_{k\in P}(x_{i}^{k}\Rightarrow(\bigvee_{l}\{(x_{l}^{k}\vee c_{lj})\wedge w_{lj}\}\vee d_{j})\geqslant\bigwedge_{k\in P}(x_{i}^{k}\Rightarrow(x_{i}^{k}\wedge w_{ij}))=\bigwedge_{k\in P}(x_{i}^{k}\Rightarrow w_{ij})\geqslant w_{ij}$. Therefore, $w_{ij}\leqslant w_{ij}^{0}$.

The following Theorem continues the previous one and shows that if there are a solution of (2) with thresholds $c_{ij}$ bounded from above by $c_{ij}^{0}$, then $w^{0}$’s, $c^{0}$’s, and $d^{0}$’s provide also a solution of (2).

Theorem 2

Let $(\textbf{W},\textbf{c},\textbf{d})\in M^{wcd}$, i.e., $M^{wcd}\neq\emptyset$, and $c_{ij}\leqslant c_{ij}^{0}$. Then $(\textbf{W}^{0},\textbf{c}^{0},\textbf{d}^{0})\in M^{wcd}$.

Proof

Let us $\forall i\in N,\;\forall j\in M,\;\forall k\in P,\;c_{ij}\leqslant c_{ij}^{0}$. Therefore, by (2), (5), (6), and Theorem 1, we get

[TABLE]

This is true because $w_{ij}^{0}\geqslant d_{j}^{0}\geqslant d_{j}$ by (7), (5), Theorem 1 and the implication Definition 8, $w_{ij}\leqslant w_{ij}^{0}$ by Theorem 1 and, hence, $(c_{ij}\vee d_{j})\wedge w_{ij}\leqslant(c_{ij}^{0}\vee d_{j}^{0})\wedge w_{ij}^{0}\leqslant y_{j}^{k}$ by (5), (6); also, $x_{i}^{k}\wedge w_{ij}^{0}\leqslant y_{j}^{k}$, since $x_{i}^{k}\wedge w_{ij}^{0}=x_{i}^{k}\wedge\bigwedge_{k\in P}(x_{i}^{k}\Rightarrow y_{j}^{k})\leqslant x_{i}^{k}\wedge(x_{i}^{k}\Rightarrow y_{j}^{k})\leqslant y_{j}^{k}$ by (7) and the implication Definition 8.

Thus, by (2), we get:

[TABLE]

Hence, $\bigvee_{i\in N}\{(x_{i}^{k}\vee c_{ij}^{0}\vee d_{j}^{0})\wedge(w_{ij}^{0}\vee d_{j}^{0})\}=y_{j}^{k}$ for $\forall j\in M,\;k\in P$, i.e., $(\textbf{W}^{0},\textbf{c}^{0},\textbf{d}^{0})\in M^{wcd}$.

The following Theorem provides us with a sufficient condition for the existence of a solution of (2).

Theorem 3

The set $M^{wcd}\neq\emptyset$ and $(\textbf{W}^{0},\textbf{c},\textbf{d})\in M^{wcd}$, if  $\forall j\in M,\;i\in N,\;c_{j}\leqslant d_{j}^{0}=\bigwedge_{k\in P}y_{j}^{k},\;\;\bigcup_{i\in N}TS_{ij}^{G}((\textbf{W}^{0},\textbf{d});Y)=P\;and\;\forall k\in P\;\bigvee_{i:\;k\in TS_{ij}^{G}}x_{i}^{k}\geqslant y_{j}^{k}$.

Proof

Let us $\forall j\in M,\;k\in P,\;\exists n\in N:\;k\in TS_{nj}^{G}((\textbf{W}^{0},\textbf{d});Y)$. Then,

[TABLE]

and

[TABLE]

Let us consider:

[TABLE]

since 11 and 12.

Hence,

[TABLE]

On the other hand, we get by the definitions of implication, $\textbf{d}^{0}$, and $\textbf{W}^{0}$ and Theorem 1 that $\forall i\in N,\;j\in M,\;k\in P,\,(x_{i}^{k}\vee c_{j}\vee d_{j})\wedge(w_{ij}^{0}\vee d_{j})\in\{x_{i}^{l}\in L\;\mid\;l\in LE_{ij}(X,Y)\}$, since $w_{ij}^{0}\geqslant d_{j}^{0}\geqslant d_{j}$, $x_{i}^{k}\wedge w_{ij}^{0}\leqslant y_{j}^{k}$ (since $x_{i}^{k}\wedge\bigwedge_{l}[x_{i}^{l}\Rightarrow y_{j}^{l}]\leqslant x_{i}^{k}\wedge[x_{i}^{k}\Rightarrow y_{j}^{k}]\leqslant y_{j}^{k}$), and $(c_{j}\vee d_{j})\leqslant y_{j}^{k}$. Hence,

[TABLE]

Combining (13) and (14), we get $(\textbf{W}^{0},\textbf{c},\textbf{d})\in M^{wcd}$, i.e., $M^{wcd}\neq\emptyset$. However, if there is a set of lattice elements that satisfy the condition of the theorem: $\{(y^{\prime})_{j}^{k}\mid y_{j}^{k}<(y^{\prime})_{j}^{k}\leqslant w_{i_{0}j}^{0}\}$ with the same matrix $w_{ij}^{0}$, the matrix is the solution only for $y$, since $(x_{i}^{k}\vee c_{j}\vee d_{j})\wedge(w_{ij}^{0}\vee d_{j})\leqslant y_{j}^{k}$ and this expression does not change for such $y^{\prime}$222$x_{i}^{k}\wedge\bigwedge_{l}[x_{i}^{l}\Rightarrow y_{j}^{l}]\leqslant x_{i}^{k}\wedge[x_{i}^{k}\Rightarrow y_{j}^{k}]=x_{i}^{k}\wedge y_{j}^{k}$. Hence, the proof is valid only for $y_{j}^{k}$.

4 Learning Algorithm

In general, all $(x_{1},y_{1}),...(x_{p},y_{p})$ can be stored in such associative memory if there exists a matrix–vector set $(\textbf{W},\textbf{c},\textbf{d})$ such as $\textbf{y}=((\textbf{x}\vee\textbf{c})\circ\textbf{W})\vee\textbf{d}$. According to Theorem 3, such sets exist if the condition of the theorem is satisfied. Therefore, to obtain solutions, it suffices to calculate expression (7) and check the corresponding condition of Theorem 3. However, such an algorithm does not demonstrate the adaptability and self-regulation of multi-valued associative memory. Therefore, we generalize the dynamic $\delta$-learning algorithm of fuzzy associative memory, introduced in Li and Ruan (1997), Li and Ruan (1999), Li and Ruan (2000) for different fuzzy cases, to the multi-valued case as in Maximov (2020b). We give this algorithm here only as illustration of adaptability capabilities of such a network since it all comes down to calculating of (7).

Let us note, that this algorithm fits only memories with the atomic lattice of weights and data, since one-to-one correspondence between lattice elements and the sets of the element generator unions exist only in this case. Also, since we use linguistic variables, not numbers, and, hence, do not use activation or membership functions, we cannot use backpropagation learning methods in our learning algorithm, unlike the advanced fuzzy case (Liu and Li (2004)). Hence, we do not use such new effective methods for parameters updating as, e.g., in Elias et al. (2020). Also, let us note that we do not have such parameters here at all.

Thus, we get an Algorithm of the memory weights $w_{ij}$ iteration for $i\in N$ and $j\in M$. We begin from the highest lattice element $w_{ij}(0)=1$ and go down until the next two iterations become equal.

Step 1.

Initialization: for $i\in N$, $j\in M$ let us put $w_{ij}(0)=1,\;t=0$;

Step 2.

Let $\textbf{W}(t)=(w_{ij}(t))$;

Step 3.

Let us calculate the resulting output of the memory (3) $\textbf{y}=((\textbf{x}\vee\textbf{c})\circ\textbf{W})\vee\textbf{d}$, i.e., $\forall k\in P,\;j\in M,\;y_{j}^{k}(t)=(\bigvee_{i}[(x_{i}^{k}\vee c_{ij})\wedge w_{ij}(t)])\vee d_{j}$. Here $y_{j}^{k}$ is the stored memory output, and $y_{j}^{k}(t)$ is its iterated output which converges to $y_{j}^{k}$ at the end of the iteration process, by (10) and Theorem 4, if a solution of (3) exists (Theorem 3).

Step 4.

Weights selection.

Denotation 1

From this place, we denote the union of generators of the lattice element $y_{j}^{k}$ by $\{y_{j}^{k}\}$ (this is a set) and the matrix of such sets $\{w\}$ corresponding to the weight matrix elements by $\{w\}_{ij}$. The matrix elements are the sets of generators of the weight matrix elements.

Matrices $w_{ij}$ of weights and $\{w\}_{ij}$ are one-to-one correspondent to each other in atomic lattices when every lattice element is represented by the join of its generators. A minus sign will denote the set difference333Given set $A$ and set $B$ the set difference of set $B$ from set $A$ is the set of all element in $A$, but not in $B$..

Then, we choose the iterated weight with the following formulas:

[TABLE]

[TABLE]

Step 5.

For $i\in N$, $j\in M$, let us check $\{w\}_{ij}(t+1)=\{w\}_{ij}(t)$? If this is true, then the Algorithm stops, otherwise $t=t+1$ and goes to Step 2.

Theorem 4

Let the matrix sequence $\{\textbf{W}(t)\;|\;t=1,2...\}$ is obtained by the learning Algorithm. Then,

• (a)

$\{\textbf{W}(t)\;|\;t=1,2...\}$* is a non-increasing sequence;*

• (b)

$\{\textbf{W}(t)\;|\;t=1,2...\}$* converges;*

• (c)

$\{\textbf{W}(t)\;|\;t=1,2...\}$* converges to $\textbf{W}^{0}$, where $w_{ij}^{0}$ is defined in (7).*

Proof

(a) For $i\in N$, $j\in M$, $k\in P$ we get from (15) that $\{w\}_{ij}^{k}(t+1)\leqslant\{w\}_{ij}^{k}(t)$. Therefore, for $i\in N$, $j\in M$: $\textbf{W}(t+1)\subseteq\textbf{W}(t)$. Thus, $\{\textbf{W}(t)\;|\;t=1,2...\}$ is a non-increasing sequence.

(b) $\{\textbf{W}(t)\;|\;t=1,2...\}$ converges, since the sequence $w_{ij}(t)$ is bounded below by the smallest lattice element 0 $\forall t=1,2...$.

(c) If, all $k$’s are such that $\forall t:\;[(x_{i}^{k}\vee c_{ij})\wedge w_{ij}(t)]\vee d_{j}\leqslant y_{j}^{k}$, then $w_{ij}(t)=w_{ij}(0)=w_{ij}^{0}=1$ by (15), (16), and Theorem 1.

In particular, if there are not (with some denotation simplification) $k\in TG_{ij}\cup NC_{ij}$, i.e., all $x_{i}^{k}\leqslant y_{j}^{k}$, then $\forall t:\;[(x_{i}^{k}\vee c_{ij}^{0})\wedge w_{ij}(t)]\vee d_{j}^{0}\leqslant y_{j}^{k}$ by (6), (5). Therefore, again, by (15), (16), and (7), $w_{ij}(t)=w_{ij}(0)=w_{ij}^{0}=1$.

If such $k$’s exist that $[(x_{i}^{k}\vee c_{ij})\wedge w_{ij}(t)]\vee d_{j}\nleqslant y_{j}^{k}$, then, we get for these $k$ on the basis of the definition of the sets’ difference operation:

$\{w\}_{ij}^{k}(1)=1-(\{x_{i}^{k}\Rightarrow\bigvee_{\begin{subarray}{c}i\in N\end{subarray}}([(x_{i}^{k}\vee c_{ij})\wedge w_{ij}(0)]\vee d_{j})\}-\{x_{i}^{k}\Rightarrow y_{j}^{k}\})=\{x_{i}^{k}\Rightarrow y_{j}^{k}\}\geqslant\{w\}_{ij}^{0}$, since the first term in parentheses is equal to 1: $\{x_{i}^{k}\Rightarrow\bigvee_{\begin{subarray}{c}i\in N\end{subarray}}([(x_{i}^{k}\vee c_{ij})\wedge w_{ij}(0)]\vee d_{j})\}=1$.

$\{w\}_{ij}^{k}(2)=\{x_{i}^{k}\Rightarrow y_{j}^{k}\}-(\{x_{i}^{k}\Rightarrow\bigvee_{\begin{subarray}{c}i\in N\end{subarray}}([(x_{i}^{k}\vee c_{ij})\wedge\{w\}_{ij}(1)]\vee d_{j})\}-\{x_{i}^{k}\Rightarrow y_{j}^{k}\})$. From here we obtain again, on the basis of the definition of the sets’ difference operation (the term in parentheses does not contain elements from all $\{x_{i}^{k}\Rightarrow y_{j}^{k}\}$):

$\{w\}_{ij}^{k}(2)=\{x_{i}^{k}\Rightarrow y_{j}^{k}\}=\{w\}_{ij}^{k}(1)$. Thus, the Algorithm converges in the second step.

It converges to $\textbf{W}^{0}$, since $w_{ij}(2)=w_{ij}(1)=\bigwedge_{k\in P}(x_{i}^{k}\Rightarrow y_{j}^{k})=w_{ij}^{0}$.

Thus, the Algorithm converges to $\textbf{W}^{0}$ and gives the solution $(\textbf{W}^{0},\textbf{c}^{0},\textbf{d}^{0})$ of (3) if such a solution exists under the conditions of Theorem 3. However, we can say nothing about existence of the solution $(\textbf{W}^{0},\textbf{c},\textbf{d})$ in general.

5 Discussion

Theorem 1 tells us that if $\textbf{W}^{0}$ is a solution, then it provides the largest solution of the equation (3), and we have used this fact in Sec. 4 when we went down from the largest lattice element to $\textbf{W}^{0}$ in a learning algorithm. The same is true for thresholds $\textbf{d}^{0}$.

However, we can say much less about thresholds c unlike the fuzzy case where similar $c^{0}$’s are the maximal possible ones, so $(\textbf{W}^{0},\textbf{c}^{0},\textbf{d}^{0})$ is the largest solution of (3) Liu and Li (2004). Indeed, we can rewrite (2) as

[TABLE]

All solutions $w_{ij}\leqslant w_{ij}^{0}=\bigwedge_{k\in P}(x_{i}^{k}\Rightarrow y_{j}^{k})\geqslant\bigwedge_{k\in P}y_{j}^{k}\leqslant c_{ij}^{0}$. Also, all $(x_{i}^{k}\wedge w_{ij}\})\leqslant y_{j}^{k}$ and $(c_{ij}\wedge w_{ij})\leqslant y_{j}^{k}$. Therefore, c may be almost arbitrary towards $\textbf{c}^{0}$.

The meaning of Theorem 3 is that not every pair of $x-y$ patterns can be stored in such associative memory with confidence (as well as in Maximov (2020a) and in a fuzzy case) – sufficiently large meets of implications in $x-y$ patterns in (7) should be enough for $\bigcup_{i\in N}TS_{ij}^{G}((\textbf{W}^{0},\textbf{d}^{0});Y)=P$. Besides, all $x$’s in every pattern cannot concentrate in the lower zone (Fig. 2), otherwise combining them may not give $y$. The storing capacity of the variant with thresholds is higher than without them, since combining c with the $x$’s in the lower zone increases them. Also, the same is true for $d$. However, we cannot assert that $c_{ij}^{0}$’s provide the largest capacity: $c_{ij}\wedge w_{ij}^{0}$ may give additional capabilities in the case of $c_{ij}$ is incomparable with $\bigwedge_{k\in P}y_{j}^{k}$ and Theorem 3 does not hold.

Finally, we will indicate without proof (the proof inLiu and Li (2004) does not change in the multi-valued case) that increasing number of neuron layers does not entail an increase of capacity.

The next point concerns the computational complexity. We have seen that implications $x_{i}^{k}\Rightarrow y_{j}^{k}$’s calculating is necessary to obtain the weight matrix in (7), (15). In the case of a complicated lattice, such a calculation may be an elaborative task: an atomic lattice with $N$ generators has $2^{N}$ elements. However, this problem is resolved in Maximov (2020b), Maximov (2020a) using a multi-valued neural network444Also, the use of a multi-valued associative memory similar to this article is proposed to obtain a control solution quickly.. Such a network has the lattice $L$ as a unique layer and a $\delta$-like activation function for a neuron in each lattice element. Such neurons skip exiting only those inputs, which may be the implication value. Then, their maximum gives the real implication. Hence, we have to obtain the maximum of at most $N-1$ nested sets of generators in order to calculate one implication. Thus, for $M$ outputs and $P$ patterns, we get at most $(N-1)^{2}NMP$ complexity. We have to add $O(P)$ to obtain meets of $k$ patterns in $w_{ij_{0}}=\bigwedge_{k\in P}(x_{i}^{k}\Rightarrow y_{j}^{k})$. Hence, computational complexity is linear by $M$ and cubic by $N$ and $P$.

There is no need to talk about cost effectiveness in our task, since we obtain an exact solution of equation (3) by the Algorithm and we do not select network parameters (we do not have them at all in the associative memory) as in, e.g., Elias et al. (2020). Hence, we can store in the memory all training patterns with confidence, if Theorem 3 holds.

6 Simulation Example

We will use a complete Brouwerian lattice $L$ as a set of values of neural network variables in the processing of control and diagnostic information of complex dynamic objects, e.g., aircraft and robotic complexes, as expanding the regular neural networks approach of

Volkov et al. (2017),

Goncharenko et al. (2016).

These authors have suggested an algorithm for functional diagnostics of such complexes, based on neural network technologies. We expand this approach to classifying spacecraft trajectories with linguistic lattice-valued estimations of trajectory parameters. Our classifying problem differs from, e.g., system state estimating in Meda-Campaca (2018) or big data classification in Hernandez et al. (2020) since we consider rather limited sets of linguistic variables which can be observed by humans (experts). Also again, we have no deal with numbers, thus our process representation is quite different. However, it seems that the equations of different estimators in Meda-Campaca (2018) may be considered with multi-valued values similarly to Maximov D. “Multi-valued Cognitive Maps”.

Thereby, let us consider a system where the state is characterized by four linguistic variables — $T_{1}...T_{4}$ — which take values in correspondent subsets of the lattice $L$ Fig. 3.

Our task is to decide which of the two classes — $C_{1}$ or $C_{2}$ — relates to the current distribution of estimates of these variables. We can imagine these variables as an alphabet of the simplest aircraft trajectories, e.g., a one-sided manoeuvre, snake manoeuvre, spiral manoeuvre, etc., and classes $C_{1}$ and $C_{2}$ as a directional movement type or a chaotic one.

Then, the lattice estimations of $T_{i},\;C_{j}$ denote degrees of confidence/truth of the correspondent valuation. However, these degrees of confidence are also, from another side, some terms in correspondent linguistic variables, e.g., the set $\{a,b,c,d\}$ may be considered as the set of directions $\{up,down,right,left\}$ diagnosed by linguistic sensors, i.e., sensors providing linguistic assessments of directions. Then, joins $ab,...$ are the moves with such possible directions’ uniting or alternating compositions. Hence, we evaluate the linguistic variables also through linguistic terms.

In the approach, the variable $T_{1}$ — e.g., horizontal snake manoeuvre — may be estimated by terms $\{ab,abc,abd\}$, the variable $T_{4}$ — e.g., vertical snake manoeuvre — may be estimated by terms $\{cd,acd,bcd\}$, the variable $T_{2}$ — spiral manoeuvre — by terms $\{ac,ad,bc,bd\}$, the variable $T_{3}$ — one-sided manoeuvre — by $\{a,b,c,d\}$, etc. The terms are really the lattice $L$ correspondent sublattices, and the sublattices’ elements evaluate them. Elements of the whole lattice $L$ estimate the classes $C_{1}$ of directional moves, and $C_{2}$ of chaotic moves.

Thus, we consider the neural network depicted in Fig. 4, in which the input variables $x_{i}$ are some valuations of the terms estimating variables $T_{i}$.

We must be able to decide which class $C_{i}$ the given object (e.g., a trajectory) belongs to if the input variables $x_{k}$ are estimated by the quantities from $T_{j}$ terms.

Let us consider the next learning pattern pair family $\{(\textbf{x}^{k},\textbf{y}^{k})\;|\;k\in P\}$ (Table 1).

Though, our lattice interpretation assumes symmetry by an $a,b,c,d$ situation, we picked up our example to demonstrate threshold using. Thus, we obtain $\textbf{c}^{0}=((c,c);(c,c);(c,c);(c,c)),\;\textbf{d}^{0}=(c,c)$ and the connection weights matrix:

[TABLE]

Such obtained thresholds mean that our training family assumes the direction to the right as a priori priority: all inputs rise up to include it.

We may easily show that the given pattern pair family $\{(\textbf{x}^{k},\textbf{y}^{k})|k\in P\}$ satisfies the conditions that $\forall j\in M,\;\bigcup_{i\in N}TS_{ij}^{G}((\textbf{W}^{0},\textbf{d}^{0});Y)=P$. Hence, by Theorem 3, all pattern pairs in Table 1 can be stored in such a multi-valued associative memory.

We may also investigate classifying properties of this neural network. Let us consider next the possible input patterns with the correspondent classification obtained (greatest output truth values determine the classification):

\begin{array}[]{c|c|c|c}k&x&y&C_{i}\\ \hline\cr 9&(abd,bd,d,d)&(bcd,cd)&C_{1}\\ \hline\cr 10&(ad,0,b,ad)&(cd,bcd)&C_{2}\\ \hline\cr 11&(bd,bd,b,a)&(bcd,bcd)&C_{1}\;or\;C_{2}\end{array}

These patterns were selected, as well as learning ones, based on our interpretation of lattice elements as aircraft manoeuvres. Thus, we see that the neural network classifies patterns quite reasonably. However, it should be noted that the system is susceptible to variations in input patterns, and it is not always possible to get such a meaningful interpretation.

In Goncharenko et al. (2016), the average probability of recognizing the type of maneuver being performed based on noisy measuring information is 0.78. In our case, noise is not essential due to roughness of linguistic assessments. However, it is difficult to estimate the recognizing possibilities of multi-valued classifier since there are currently no real data associated with the multi-value patterns. We can only establish that al training patterns are recognized and that other patterns may take incomparable estimations. Hence, we may make a definite decision not always.

7 Conclusion

This study introduces a new concept of multi-valued neural networks with thresholds, in which weights and data represent not by numbers. The idea continues our studies in which the state of a system is estimated not by numbers, but by elements of a partially ordered set, namely, the lattice. In our case, the lattice is finite and distributive, and it consists of some linguistic variables. Such an approach can facilitate a situation which requires the participation of expert evaluation: in this case — assessments are the linguistic variables, not numbers. Thus, they do not require vague concepts of defuzzification.

We have expanded the results on fuzzy neural networks to such a multi-valued case in which variables take value in a Brouwer lattice. We discovered conditions under which it is possible to store given pairs of linguistic patterns in such a multi-valued associative memory. Here, thresholds increase the network storing capacity as in the fuzzy case. We also gave the learning algorithm generalizing the fuzzy one without thresholds. Finally, we gave the example of the network use to classifying aircraft/spacecraft trajectories’.

However, the classification method has an obvious limitation arising from the nature of the representation of variable: our classifier does not always distinguish the system states since they may be incomparable (as the lattice elements). Also, we were not able to get the solutions with the highest capacity. We have obtained only the condition of existence of the solution with largest weights and certain thresholds.

Future researches are going in two directions. First, we develop multi-valued cognitive maps which have the neural network nature. Second, we suppose to expand use of a lattice elements as weights and data values to the lattice subsets, as analogue of fuzzy numbers. We assume such an expansion will give an opportunity for the networks use in more complicated cases, e.g., in image restoration.

Conflicts of interest

The authors declare that they have no conflict of interest.