A Broad Class of Discrete-Time Hypercomplex-Valued Hopfield Neural Networks

TL;DR

This paper introduces real-part associative hypercomplex number systems and $$-projection activation functions to analyze and ensure the stability of a broad class of discrete-time hypercomplex-valued Hopfield neural networks, generalizing existing models.

Contribution

It proposes novel hypercomplex number systems and activation functions, extending stability analysis to a wide class of hypercomplex-valued neural networks including Cayley-Dickson algebras.

Findings

Confirmed stability of several existing hypercomplex neural networks

Introduced a general stability framework for Cayley-Dickson algebra-based networks

Extended stability analysis to new hypercomplex systems

Abstract

In this paper, we address the stability of a broad class of discrete-time hypercomplex-valued Hopfield-type neural networks. To ensure the neural networks belonging to this class always settle down at a stationary state, we introduce novel hypercomplex number systems referred to as real-part associative hypercomplex number systems. Real-part associative hypercomplex number systems generalize the well-known Cayley-Dickson algebras and real Clifford algebras and include the systems of real numbers, complex numbers, dual numbers, hyperbolic numbers, quaternions, tessarines, and octonions as particular instances. Apart from the novel hypercomplex number systems, we introduce a family of hypercomplex-valued activation functions called $\mathcal{B}$-projection functions. Broadly speaking, a $\mathcal{B}$-projection function projects the activation potential onto the set of all possible states of a hypercomplex-valued neuron. Using the theory presented in this paper, we confirm the stability analysis of several discrete-time hypercomplex-valued Hopfield-type neural networks from the literature. Moreover, we introduce and provide the stability analysis of a general class of Hopfield-type neural networks on Cayley-Dickson algebras.

Tables

Table 1. Table 1: Some multiplication tables

a) Complex numbers	b) Hyperbolic numbers	c) Dual numbers
$\times$ $𝐢$ $𝐢$ $-1$	$\times$ $𝐢$ $𝐢$ $+1$	$\times$ $𝐢$ $𝐢$ $0$

Equations

$$p=p_0+p_1{\mathbf{i}}_1+\dots +p_n{\mathbf{i}}_n,$$

$$p+q=(p_0+q_0)+(p_1+q_1){\mathbf{i}}_1+\dots +(p_n+q_n){\mathbf{i}}_n.$$

$${\mathbf{i}}_{\mu }{\mathbf{i}}_{\nu }=a_{\mu \nu ,0}+a_{\mu \nu ,1}{\mathbf{i}}_1+\dots +a_{\mu \nu ,n}{\mathbf{i}}_n,\forall \mu ,\nu \in \{1,\dots ,n\}.$$

$$pq=\left(p_0{q}_0+\sum _{\mu ,\nu =1}^n{p}_{\mu }{q}_{\nu }{a}_{\mu \nu ,0}\right)$$

$$+\left(p_0{q}_n+p_n{q}_0+\sum _{\mu ,\nu =1}^n{p}_{\mu }{q}_{\nu }{a}_{\mu \nu ,n}\right){\mathbf{i}}_n.$$

$$\displaystyle\tau\big{(}\tau(p)\big{)}=p,\hfill$$

$$\tau(p)=\tau\big{(}p_{0}+\vec{p}\big{)}=\tau\big{(}p_{0}\cdot 1+\vec{p}\big{)}=p_{0}\tau(1)+\tau(\vec{p})=p_{0}+\tau(\vec{p}).$$

$$\text{Re}\left\{\tau (p)\right\}=\text{Re}\left\{p\right\},\forall p\in \mathbb{H}.$$

$$\bar{p}=p_0-\vec{p}.$$

$$\mathcal{B}(p,q)=\text{Re}\left\{\tau (p)q\right\},\forall p,q\in \mathbb{H}.$$

$$\mathcal{B}(p,q){=}^{\text{eqref}eq:inner-product}\text{Re}\left\{\tau (p)q\right\}{=}^{\text{eqref}eq:realtau}\text{Re}\left\{\tau (\tau (p)q)\right\}{=}^{\text{eqref}eq:nu3}\text{Re}\left\{\tau (q)\tau (\tau (p))\right\}{=}^{\text{eqref}eq:nu1}\text{Re}\left\{\tau (q)p\right\}{=}^{\text{eqref}eq:inner-product}\mathcal{B}(q,p).$$

$$\text{Re}\left\{(pq)r-p(qr)\right\}=0.$$

$$\mathcal{B}(pq,r)=\mathcal{B}(q,\tau (p)r).$$

$$\tau (p)=p_0-p_1\mathbf{i}+p_2\mathbf{j}-p_3\mathbf{k},\forall p\in \mathbb{T},$$

$$(pq)r=p(qr)\text{mbox}and{p}^2\in \mathbb{R},\forall p,q,r\in Cl(\mathbb{R}).$$

$$\overline{(x_1,y_1)}=({\bar{x}}_1,-y_1),$$

$$(x_1,y_1)(x_2,y_2)=(x_1{x}_2-y_2{\bar{y}}_1,{\bar{x}}_1{y}_2+x_2{y}_1).$$

$$\text{Re}\left\{(x,y)\right\}=\text{Re}\left\{x\right\},\forall (x,y)\in A_{k+1},$$

$$\mathcal{B}\big{(}(x_{1},y_{1}),(x_{2},y_{2})\big{)}=\mathcal{B}(x_{1},x_{2})+\mathcal{B}(y_{1},y_{2})$$

$$\mathcal{B}(p,q)=\sum _{i=0}^n{p}_i{q}_i,$$

$$x_{i}(t+\Delta t)=\begin{cases}f\big{(}v_{i}(t)\big{)},&v_{i}(t)\in\mathcal{D},\\ x_{i}(t),&\mbox{otherwise},\end{cases}$$

$$\mathcal{B}(f(q),q)>\mathcal{B}(s,q),\forall q\in D,\forall s\in \mathcal{S}\setminus \left\{f(q)\right\}.$$

$$E(\mathbf{x})=-\frac{1}{2}\sum _{i=1}^N\sum _{j=1}^N\text{Re}\left\{\tau (x_i)(w_{ij}{x}_j)\right\},$$

$$E(\mathbf{x})=-\frac{1}{2}\sum _{i=1}^N\sum _{j=1}^N\mathcal{B}(x_i,w_{ij}{x}_j).$$

$$\Delta E=E(\mathbf{x}(t+\Delta t))-E(\mathbf{x}(t))<0,$$

$$\mathcal{B}(z_1,z_2)=|z_1||z_2|\cos |{\theta }_1-{\theta }_2|,$$

$$\text{csgn}(z)=\{\begin{array}{ll}1,& 0\le \arg (z)<\Delta \theta ,\\ e^{2\mathbf{i}\Delta \theta },& \Delta \theta <\arg (z)<3\Delta \theta ,\\ ⋮& ⋮\\ 1,& (2K-1)\Delta \theta <\arg (z)<2\pi \end{array}$$

$$\mathcal{D}=\{z\in \mathbb{C}\setminus \{0\}:\arg (z)\ne (2k-1)\Delta \theta ,\forall k=1,\dots ,K\},$$

$$\mathcal{S}=\{1,e^{2\mathbf{i}\Delta \theta },e^{4\mathbf{i}\Delta \theta },\dots ,e^{2(K-1)\mathbf{i}\Delta \theta }\}$$

$$|\theta -\alpha |<\Delta \theta <|\theta -\beta |.$$

$$\mathcal{B}(\text{csgn}(z),z)=|z|\cos |\theta -\alpha |>|z|\cos |\theta -\beta |=\mathcal{B}(s,z).$$

Full text

Copyright Notice

Personal use of this material is permitted. Permission from Elsevier must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.

Neural Networks. Volume 122, February 2020, Pages 54-67. https://doi.org/10.1016/j.neunet.2019.09.040

1 Introduction

Hopfield-type neural networks (HNNs) are important recurrent neural networks that can be used to implement associative memories [18]. Besides implementing associative memories, HNNs have been applied in control [12, 56], computer vision and image processing [65, 45], classification [51, 67], and optimization [19, 53, 44].

Although the HNN has been originally conceived for bipolar state neurons [18], it has been extended to hypercomplex-valued neurons using complex numbers [25], dual numbers [42], hyperbolic numbers [27], tessarines [21], quaternions [20], octonions [41], and further hypercomplex number systems [63, 40, 52]. Extensions of the traditional HNN using hypercomplex numbers are referred to as hypercomplex-valued Hopfield-type neural networks (HHNNs). In this paper, we focus on discrete-time HHNNs.

Hypercomplex-valued neural networks can treat many kinds of information which may not be properly captured by real-valued neural networks, such as, phase, tensors, spinors, and multidimensional geometrical affine transformations [11, 48, 16]. Moreover, in contrast to real-valued neural networks, hypercomplex-valued neural networks naturally treat multidimensional data as single entities. In other words, a hypercomplex-valued neural network can cope in an almost natural way with multidimensional data while constraints must be imposed on the topology of a real-valued neural network to take advantage of the correlation between the variables. In fact, a certain complex-valued neural network with a single hidden layer using frequency domain features outperformed the best real-valued neural network models in an image recognition task [1]. Quaternion-valued neural networks outperformed real-valued networks for color and PolSAR image processing tasks [46, 54, 26]. Quaternion-valued neural networks also outperformed real-valued models for three- and four-dimensional time series prediction as well as target tracking [13, 38, 66, 57].

Briefly, an HHNN can be classified according to the activation function of its neurons. For example, a split HHNN is derived by applying a real-valued function to each component of the hypercomplex-valued activation potential of a neuron. Also, the target set of an activation function equals the set of all possible states of a hypercomplex-valued neuron. Accordingly, a multistate HHNN is obtained by considering a finite target set [25, 20, 22] while a continuous target set yields a continuous-valued HHNN [2, 49, 50, 60, 62].

1.1 A Short Literature Review on HHNNs

Although research on discrete-time HHNN dates to the late 1980s [2, 49, 50], in our opinion, the most relevant contribution is the complex-valued multistate model proposed by [25]. The output of a multistate complex-valued neuron is obtained by applying the complex-valued signum function (csgn) on the activation potential of a neuron. The csgn function, which has been introduced by [2], yields the root of the complex unit which is obtained by quantizing the phase of its argument. [25] have affirmed that their multistate complex-valued model with asynchronous update always settles down at a stationary state under the usual conditions on the synaptic weights, that is, Hermitian connections ($w_{ij}=\bar{w}_{ji}$), and non-negative self-feedback ($w_{ii}\geq 0$). However, [68] showed that the model of Jankowski et al. may fail to yield a convergent sequence if $w_{ii}=0$.

In $2008$, Isokawa et al. extended the discrete-time multistate complex-valued neural network of Jankowski et al. to quaternions [22, 23]. Briefly, a quaternionic version of the complex-valued signum function, denoted by qsgn, is used in their model. The quaternionic signum function yields the unit quaternion obtained by quantizing the three phase-angles of its quaternionic argument. In $2016$, Minemoto et al. introduced a slight modification of the quaternionic multistate HNN of Isokawa et al. which is numerically stable [46, 61]. Despite the successful application for the reconstruction of color images, the HHNN of Minemoto et al. does not always settle down at an equilibrium state [62, 61]. Nevertheless, using a limit process, we obtain a discrete-time continuous-valued quaternionic HNN on unit quaternions which always comes to rest at an equilibrium state [60, 62].

Apart from the extensions of the discrete-time HNN using complex numbers and quaternions, HHNNs on hyperbolic and dual numbers have been proposed and investigated by [27, 29, 28, 35, 34]. Also, [6] introduced a discrete-time continuous-valued octonionic HHNN. HHNNs on tessarines, which are also referred to as commutative quaternions, have been investigated by [21] and, more recently, by [36].

1.2 Contributions and Organization of the Paper

It turns out that complex numbers, quaternions, and octonions are, apart from an isomorphism, all instances of Cayley-Dickson algebras. Complex numbers and quaternions are also isomorphic to instances of real Clifford algebras. In view of these remarks, this paper aims to provide a unifying framework for HHNNs defined on general hypercomplex number systems.

Precisely, we introduce novel hypercomplex number systems which generalize the Cayley-Dickson and real Clifford algebras but enjoy the main properties for the stability analysis of HHNNs. The novel systems, referred to as real-part associative hypercomplex number systems, provide one of the most general hypercomplex number systems for which a Hopfield-type neural network can be defined and properly analyzed. We also introduce a broad family of hypercomplex-valued activation functions and provide an important theorem concerning the stability (in the sense of Lyapunov) for discrete-time hypercomplex-valued Hopfield-type neural networks. In fact, the theorem presented in this paper can be applied for the stability analysis of many discrete-time HHNNs from the literature, including complex-valued [25, 68, 32], hyperbolic-valued [27, 29], dual-numbered [34], tessarine-valued [21, 36], quaternion-valued [20, 62], and octonion-valued Hopfield neural networks [6].

In particular, this paper contributes with a slight modification of the complex-valued multistate Hopfield neural network of Jankowski et al. which always settle down at an equilibrium state under the usual conditions on the synaptic weights, including $w_{ii}=0$. In other words, using the theory developed for a broad class of discrete-time hypercomplex-valued Hopfield-type neural networks, in this paper we present a solution to an open problem in the literature. Moreover, we address the stability of complex-valued, hyperbolic-valued, and dual-numbered Hopfield-type neural networks with the split-sign function and we point out some interesting relations between them as well as their corresponding real-valued (bipolar) models. We also introduce a broad class of discrete-time Hopfield-type neural networks defined on Cayley-Dickson algebras which include the complex-valued, quaternion-valued, and octonion-valued models as particular instances. The stability analysis of the broad class of Cayley-Dickson Hopfield-type neural networks follows from the theory presented in this paper.

This paper is organized as follows: Next section reviews the main concepts on hypercomplex numbers. The real-part associative hypercomplex number systems are also introduced in Section 2. In Section 3, we introduce a broad class of activation functions and present a theorem concerning the stability of HHNNs. In Section 4, we apply the theory introduced previously for the stability analysis of several HHNNs from the literature. The modification of the complex-valued multistate Hopfield neural network of Jankowski et al. and the novel HHNNs on Cayley-Dickson algebras are presented in Section 4. Concluding remarks are given in Section 5.

2 Hypercomplex numbers

A number $p$ is said hypercomplex when it can be represented in the form

[TABLE]

where $n$ is a non-negative integer, $p_{0},p_{1},\ldots,p_{n}$ are real numbers, and the symbols $\mathbf{i}_{1},\mathbf{i}_{2},\ldots,\mathbf{i}_{n}$ (written using boldface in this paper) are called hyperimaginary units [55]. We denote by $\mathbb{H}$ the set (universe) of all hypercomplex numbers given by (1). Examples of hypercomplex numbers include real numbers, complex numbers, hyperbolic numbers, dual numbers, quaternions, tessarines (also called commutative quaternions), and octonions, denoted in this paper respectively by $\mathbb{R}$, $\mathbb{C}$, $\mathbb{U}$, $\mathbb{D}$, $\mathbb{Q}$, $\mathbb{T}$, and $\mathbb{O}$.

Note that a hypercomplex number $p={p}_{0}+{p}_{1}\mathbf{i}_{1}+\ldots+{p}_{n}\mathbf{i}_{n}$ can be identified with the $(n+1)$-tuple $(p_{0},p_{1},\ldots,p_{n})$ of real numbers. Thus, we say that the dimension of $\mathbb{H}$ is $n+1$ and write $\mathtt{dim}(\mathbb{H})=n+1$. For example, $\mathtt{dim}(\mathbb{R})=1$, $\mathtt{dim}(\mathbb{C})=\mathtt{dim}(\mathbb{U})=\mathtt{dim}(\mathbb{D})=2$, $\mathtt{dim}(\mathbb{Q})=\mathtt{dim}(\mathbb{T})=4$, and $\mathtt{dim}(\mathbb{O})=8.$ Furthermore, the set $\mathbb{H}$ of all hypercomplex numbers inherits the topology from $\mathbb{R}^{n+1}$. For instance, we say that $\mathcal{S}\subseteq\mathbb{H}$ is compact if and only if the set $\{(p_{0},p_{1},\ldots,p_{n}):{p}_{0}+{p}_{1}\mathbf{i}_{1}+\ldots+{p}_{n}\mathbf{i}_{n}\in\mathcal{S}\}\subseteq\mathbb{R}^{n+1}$ is compact.

A hypercomplex number system is a set of hypercomplex numbers equipped with an addition and a multiplication (or product). The addition of two hypercomplex numbers $p={p}_{0}+{p}_{1}\mathbf{i}_{1}+\ldots+{p}_{n}\mathbf{i}_{n}$ and $q={q}_{0}+{q}_{1}\mathbf{i}_{1}+\ldots+{q}_{n}\mathbf{i}_{n}$ is defined in a component-wise manner according to the expression

[TABLE]

The product between $p$ and $q$, denoted by the juxtaposition of $p$ and $q$, is defined as follows: First, we assign to each product of two hyperimaginary units $\mathbf{i}_{\mu}$ and $\mathbf{i}_{\nu}$, $\mu,\nu\in\{1,\ldots,n\}$, a new hypercomplex number. Mathematically, we define

[TABLE]

We would like to point out that (3) determines a multiplication table. Precisely, the product between $\mathbf{i}_{\mu}$ and $\mathbf{i}_{\nu}$ is the hypercomplex number situated in the intersection of the $\mu$th row and the $\nu$th column in this table. For instance, Table 1 shows the multiplication table of the complex numbers, hyperbolic numbers, dual numbers, quaternions, and tessarines. For simplicity, we write $\mathbf{i}_{1}=\mathbf{i},\mathbf{i}_{2}=\mathbf{j}$, and $\mathbf{i}_{3}=\mathbf{k}$.

The product $pq$ between the hypercomplex numbers $p$ and $q$ is determined using the distributive law and the multiplication table as follows. Each term $(p_{\mu}\mathbf{i}_{\mu})(q_{\nu}\mathbf{i}_{\nu})$ is rewritten as $p_{\mu}q_{\nu}(\mathbf{i}_{\mu}\mathbf{i}_{\nu})$ and the product $\mathbf{i}_{\mu}\mathbf{i}_{\nu}$ is replaced in accordance with (3). Formally, the product is given by

[TABLE]

Note that we can identify a real number $\alpha\in\mathbb{R}$ with the hypercomplex number $\alpha+0\mathbf{i}_{1}+\ldots+0\mathbf{i}_{n}\in\mathbb{H}$. Hence, the scalar multiplication can be viewed as a particular case of the hypercomplex multiplication given by (2).

The addition and the multiplication, given respectively by (2) and (2), satisfy the following properties for all $\alpha,\beta\in\mathbb{R}$ and $p,q,r\in\mathbb{H}$:

$\alpha p=\alpha p_{0}+(\alpha p_{1}){\mathbf{i}_{1}}+\ldots+(\alpha p_{n}){\mathbf{i}_{n}}=p\alpha$. 2. 2.

$(\alpha p)(\beta q)=(\alpha\beta)(pq)$. 3. 3.

$p(q+r)=pq+pr$ and $(p+q)r=pr+qr$.

Many other properties of multiplication, such as commutativity and associativity, do not necessarily hold true. Indeed, the multiplication between tessarines is commutative and associative; the multiplication between quaternions is associative but it is not commutative; the multiplication between octonions is neither commutative nor associative.

A hypercomplex number $p$ can also be written as $p=p_{0}+\vec{p}$, where $p_{0}$ and $\vec{p}=p_{1}\mathbf{i}_{1}+\ldots+p_{n}\mathbf{i}_{n}$ are called, respectively, the real and the vector parts of $p$. We denote the real part of $p$ by $\text{Re}\left\{p\right\}:=p_{0}$ and its vector part by $\text{Ve}\left\{p\right\}:=\vec{p}$.

Borrowing the terminology of linear algebra, we speak of a linear operator $T:\mathbb{H}\to\mathbb{H}$ if $T(\alpha p+q)=\alpha T(p)+T(q)$, for all $p,q\in\mathbb{H}$ and $\alpha\in\mathbb{R}$. A linear operator that is an involution and also an antihomomorphism is called a reverse-involution [10], and it is formally defined as follows:

Definition 1** (Reverse-involution).**

An operator $\tau:\mathbb{H}\to\mathbb{H}$ is a reverse-involution if

[TABLE]

for all $p,q\in\mathbb{H}$ and $\alpha\in\mathbb{R}$.

From (7) and (7), we conclude that $1\equiv 1+0\mathbf{i}_{1}+\ldots+0\mathbf{i}_{n}$ is a fixed point of a reverse-involution $\tau$, that is, the equation $\tau(1)=1$ holds true. Furthermore, by writing $p=p_{0}+\vec{p}$ and using the linearity property (7), we conclude that

[TABLE]

Therefore, we derive the important identity:

[TABLE]

The natural conjugation is an example of a reverse-involution in some hypercomplex number systems such as the complex and quaternion number systems. Formally, the natural conjugate of a hypercomplex number $p$ is denoted by $\bar{p}$ and is defined by

[TABLE]

Other examples of reverse-involutions include the quaternion anti-involutions in quaternion algebra and the Clifford conjugation in Clifford algebras [10, 9, 64]. Also, if the multiplication is commutative, then the identity mapping $\tau(p)=p$, for all $p\in\mathbb{H}$, is referred to as the trivial reverse-involution.

Finally, let us define the symmetric bilinear form $\mathcal{B}:\mathbb{H}\times\mathbb{H}\to\mathbb{R}$ by means of the following equation:

[TABLE]

Remark 1**.**

The linearity in the first argument of $\mathcal{B}$ follows from (7) while the following shows that it is also symmetric:

[TABLE]

Intuitively, $\mathcal{B}$ measures a relationship between $p$ and $q$ by taking into account the algebraic properties of the multiplication and the reverse-involution $\tau$. For example, the symmetric bilinear form $\mathcal{B}$ coincides with the usual inner product on complex numbers, quaternions, and octonions with the natural conjugation.

2.1 Real-Part Associative Hypercomplex Number Systems

In general, the mathematical properties presented in the previous section hold true for an arbitrary hypercomplex number system $\mathbb{H}$. Let us now define a broad class of hypercomplex number systems with a reverse-involution that enjoy an extra property: associativity holds in the real-part of the product [5].

Definition 2** (Real-Part Associative Hypercomplex Number Systems).**

A hypercomplex number system equipped with a reverse-involution $\tau$ is called a real-part associative hypercomplex number system (Re-AHN) if the following identity holds true for any three of its elements $p,q,r$:

[TABLE]

In particular, we speak of a positive semi-definite (or non-negative definite) real-part associative hypercomplex number system if the symmetric bilinear form $\mathcal{B}$ given by (10) satisfies $\mathcal{B}(p,p)\geq 0$, $\forall p\in\mathbb{H}$.

We would like to point out that the identity (11) has been used implicitly for the stability analysis of many HHNN models [25, 22, 60]. Furthermore, this property is used explicitly when the product is not associative, for instance, in the stability analysis of octonion-valued Hopfield-type neural networks [41, 6]. Finally, since the reverse-involution plays an important role for the stability analysis of HHNNs, it has been included in the definition of a real-part associative hypercomplex number system.

Remark 2**.**

The element $(pq)r-p(qr)$ is called associator of the real-part associative hypercomplex number systems relative to the product and measures its degree of non-associativity. Evidently, (11) holds true if the multiplication is associative.

Remark 3**.**

In a real-part associative hypercomplex number system, the following identity holds true for any $p,q,r\in\mathbb{H}$:

[TABLE]

We can prove by direct computation that the systems of real numbers, complex numbers, quaternions, and octonions are real-part associative hypercomplex number systems with the natural conjugation. More generally, real-part associative hypercomplex number systems include the tessarines, Cayley-Dickson algebras, and real Clifford algebras.

Example 1**.**

The set of tessarines, also known as commutative quaternions and denoted by $\mathbb{T}$, is composed by hypercomplex numbers of the form $p=p_{0}+p_{1}\mathbf{i}+p_{2}\mathbf{j}+p_{3}\mathbf{k}$ whose multiplication is given by Table 1e). The reader interested on a historical account on the emergence of tessarines is invited to consult [8]. The tessarines algebra is associative and, thus, the identity $\text{Re}\left\{(pq)r\right\}=\text{Re}\left\{p(qr)\right\}$ holds true for all $p,q,r\in\mathbb{T}$. Moreover, the reverse-involution given by

[TABLE]

yields, by means of (10), a symmetric bilinear form such that $\mathcal{B}(p,p)=p_{0}^{2}+p_{1}^{2}+p_{2}^{2}+p_{3}^{2}\geq 0,\forall p\in\mathbb{T}$. Hence, $\mathbb{T}$ is a positive semi-definite real-part associative hypercomplex number system with the reverse-involution given by (13).

Example 2**.**

Real Clifford algebras ($Cl(\mathbb{R})$), also referred to as geometric algebras, can be seen as hypercomplex number systems that incorporate the geometric notion of direction and orientation [9, 64, 15]. Roughly speaking, a real Clifford algebra can be seen as a hypercomplex number system of dimension $n=2^{k}$ in which the product is associative and the square of an element is always a scalar, that is,

[TABLE]

The reader interested in the geometric interpretation as well as the formal definition of Clifford algebras is invited to consult [9, 64, 15]. Also, a detailed survey on the applications of Clifford algebras with focus on neurocomputing and correlated areas can be found in [17]. In order to maintain geometric notions, the Clifford conjugation is a reverse-involution [9, 64]. Furthermore, since the identity (11) holds true in an associative hypercomplex number system, we conclude that real Clifford algebras equipped with the Clifford conjugation are real-part associative hypercomplex number systems. Complex, hyperbolic, and dual numbers are all isomorphic to examples of 2-dimensional real Clifford algebras. Apart from an isomorphism, quaternion algebra is an example of a 4-dimensional real Clifford algebra.

Example 3**.**

According to [4], Cayley-Dickson algebras are finite-dimensional real algebras defined recursively as follows: The first Cayley-Dickson algebra, denoted by $A_{0}$, is the real number system. Given a Cayley-Dickson algebra $A_{k}$, the next algebra $A_{k+1}$ comprises all pairs $(x,y)\in A_{k}\times A_{k}$ with the component-wise addition. The conjugation and product are defined as follows for any $(x_{1},y_{1}),(x_{2},y_{2})\in A_{k+1}$:

[TABLE]

and

[TABLE]

The real part is also defined recursively by setting $\text{Re}\left\{x\right\}=x$ for all $x\in A_{0}$ and

[TABLE]

where the term on the right-hand side denotes the real part of $x\in A_{k}$. Also, the symmetric bilinear form $\mathcal{B}$ given by (10) satisfies $\mathcal{B}(x,x)=x^{2}$ for all $x\in A_{0}$ and

[TABLE]

for all $(x_{1},y_{1}),(x_{2},y_{2})\in A_{k+1}.$

It is not hard to verify that $A_{1}$, $A_{2}$, and $A_{3}$ correspond respectively to the algebras of complex numbers, quaternions, and octonions. Furthermore, we can identify a Cayley-Dickson algebra $A_{k}$ with a positive semi-definite real-part associative hypercomplex number systems with dimension $2^{k}$. In fact, identity (11) is a consequence of Lemma 2.8 from [4]. By induction, it is not hard to show that the conjugation defined by (15) corresponds to the natural conjugation given by (9). Also, the natural conjugation is a reverse-involution on $A_{k}$. Finally, from (18), the symmetric bilinear form $\mathcal{B}:A_{k}\times A_{k}\to\mathbb{R}$ satisfies

[TABLE]

for all $p={p}_{0}+{p}_{1}\mathbf{i}_{1}+\ldots+{p}_{n}\mathbf{i}_{n}$ and $q={q}_{0}+{q}_{1}\mathbf{i}_{1}+\ldots+{q}_{n}\mathbf{i}_{n}$ with $n=2^{k}-1$. Thus, $\mathcal{B}(p,p)=\sum_{i=0}^{2^{k}-1}p_{i}^{2}\geq 0$, which implies that the Cayley-Dickson algebra $A_{k}$ is a positive semi-definite real-part associative hypercomplex number system.

Finally, the diagram shown in Figure 1 illustrates the inclusion relationships between some hypercomplex number systems.

3 Hypercomplex Hopfield Neural Networks

Let $\mathbb{H}$ be a real-part associative hypercomplex number system and $\mathcal{S}\subset\mathbb{H}$ be the set of all possible states of a hypercomplex-valued neuron. Like the traditional discrete-time HNN, a discrete-time HHNN is a recurrent neural network with $N$ hypercomplex-valued neurons. Let $x_{i}(t)\in\mathcal{S}$ denote the hypercomplex-valued state of the $i$th neuron at time $t\geq 0$, for $i=1,\ldots,N$. Also, let $w_{ij}\in\mathbb{H}$ be the $j$th hypercomplex-valued synaptic weight of the $i$th neuron. Given an initial hypercomplex-valued state vector $\mathbf{x}(0)=[x_{1}(0),\ldots,x_{N}(0)]^{T}\in\mathcal{S}^{N}$, the HHNN defines recursively the sequence $\{\mathbf{x}(t)\}_{t\geq 0}$ by means of the equation

[TABLE]

where $v_{i}(t)=\sum_{j=1}^{N}w_{ij}x_{j}(t)$ is the hypercomplex-valued activation potential of the $i$th neuron at time $t$, and $f$ is a hypercomplex-valued activation function with domain $\mathcal{D}\subset\mathbb{H}$ and codomain $\mathcal{S}\subset\mathbb{H}$. Note that the $i$th neuron remains in its state if the activation function is not defined at the hypercomplex-valued activation potential, that is, we have $x_{i}(t+\Delta t)=x_{i}(t)$ if $v_{i}(t)\not\in\mathcal{D}$.

Broadly speaking, the activation function $f:\mathcal{D}\to\mathcal{S}$ in (20) should project the activation potential onto the set of all possible states of the hypercomplex-valued neuron. In other words, we expect $f(q)$ to be more related to $q\in\mathcal{D}$ than any other element $s\in\mathcal{S}$. Furthermore, this relationship between $f(q)$ and $q$ should take into account the algebraic structure of the real-part associative hypercomplex number system. Broadly speaking, $f$ should project $q$ onto $S$ with respect to the bilinear quadract form $\mathcal{B}$, which takes into account both the product and the reverse-involution. The following class of activation functions, which has been introduced by [5], formalizes these remarks.

Definition 3** ($\mathcal{B}$-projection function).**

Consider a real-part associative hypercomplex number system $\mathbb{H}$ equipped with a reverse-involution $\tau$ and let $\mathcal{D}\subset\mathbb{H}$, $\mathcal{S}\subset\mathbb{H}$, and $\mathcal{B}:\mathbb{H}\times\mathbb{H}\to\mathbb{R}$ be the symmetric bilinear form defined by (10). A hypercomplex-valued function $f:\mathcal{D}\to\mathcal{S}$ is called a $\mathcal{B}$-projection function if

[TABLE]

Examples of $\mathcal{B}$-projection functions include the complex-valued signum function for complex-valued Hopfield neural networks, the split-sign and the function that normalizes its arguments to length one on Cayley-Dickson algebras with the natural conjugation, and the split-sign functions for some real Clifford algebras with Clifford conjugation. Examples of $\mathcal{B}$-projection functions are addressed in details in Section 4. The multistate quaternion-valued signum function introduced by [20] using quantizations of the phase-angles is not a $\mathcal{B}$-projection function. At this point, however, we would like to call the reader’s attention to the following facts: First, note that $0\equiv 0+0\mathbf{i}_{1}+\ldots+0\mathbf{i}_{n}$ cannot belong to the domain $D$ of a $\mathcal{B}$-projection function because $\mathcal{B}(p,0)=0$ for all $p\in\mathbb{H}$. Thus, the condition given by (21) does not hold true if $0\in D$. Second, but not less important, the inequality in (21) depends not only on the expression of $f$ but also on the reverse-involution and the product of hypercomplex numbers (by means of the symmetric bilinear form).

3.1 Stability Analysis of HHNNs

In many applications, including the implementation of associative memories and solving optimization problems, we are interested in the convergence of the sequence $\{\mathbf{x}(t)\}_{t\geq 0}$ generated by an HHNN. In this subsection, we address this important issue. Precisely, we will study the dynamic of an HHNN by means of the energy function $E$ defined by

[TABLE]

where $N$ is the number of neurons of the neural network, $w_{ij}$ denotes the $j$th synaptic weight of the $i$th hypercomplex-valued neuron, and $\tau$ is the reverse-involution of a real-part associative hypercomplex number system $\mathbb{H}$. Alternatively, we can express the energy function $E$ as follows using the symmetric bilinear form $\mathcal{B}$ given by (10):

[TABLE]

The convergence of the sequence $\{\mathbf{x}(t)\}_{t\geq 0}$ defined by (20) is ensured by showing that $E$ given by (22) is real-valued, bounded and decreasing along any non-stationary trajectory, i.e, the inequality

[TABLE]

holds true whenever $\mathbf{x}(t+\Delta t)\neq\mathbf{x}(t)$.

Theorem 1, whose proof can be found in the appendix, addresses the convergence of the sequence produced by (20) with a $\mathcal{B}$-projection function $f$.

Theorem 1**.**

Let $f:\mathcal{D}\to\mathcal{S}$ be a $\mathcal{B}$-projection function and $\mathbf{x}(0)\in\mathcal{S}^{N}$, where $\mathcal{S}$ is a compact subset of a real-part associative hypercomplex number system $\mathbb{H}$. The sequence produced by (20) is convergent, in an asynchronous update mode, if the synaptic weights satisfy $w_{ij}=\tau(w_{ji})$ and one of the two cases below holds true:

$w_{ii}=0$ for any $i\in\{1,\ldots,N\}$. 2. 2.

$w_{ii}$ is a nonnegative real number for any $i\in\{1,\ldots,N\}$ and $\mathbb{H}$ is a positive semi-definite real-part associative hypercomplex number system.

4 Examples

In this section we present examples illustrating that the theory presented in this paper generalizes several results from the literature concerning the stability analysis of HHNNs. We also use the theory presented above to introduce two families of Hopfield-type networks on Cayley-Dickson algebras (cf. Subsection 4.5).

4.1 Bipolar Hopfield Neural Network

Consider the system of real numbers $\mathbb{R}$ equipped with the trivial reverse-involution, i.e., the identity operator $\tau(x)=x$ for all $x\in\mathbb{R}$. Note that $\mathbb{R}$ with the identity operator is a positive semi-definite real-part associative hypercomplex number system because $\mathcal{B}(x,y)=xy$ and $\mathcal{B}(x,x)=x^{2}\geq 0$.

Apart from an isomorphism, the famous discrete-time real-valued recurrent neural network introduced by [18] is given by (20) with the activation function $f\equiv\text{sgn}:\mathcal{D}\to\mathcal{S}$, where the domain $\mathcal{D}=\mathbb{R}\setminus\{0\}$ is the set of all non-zero real numbers and $\mathcal{S}=\{-1,+1\}$ is the set of all possible states of a real-valued bipolar neuron. The function $\text{sgn}:\mathcal{D}\to\mathcal{S}$, which can be expressed as $\text{sgn}(x)=x/|x|$, is a $\mathcal{B}$-projection function. In fact, for any $x\in\mathcal{D}$, we have $\mathcal{B}(\text{sgn}(x),x)=|x|$ while $s\in\mathcal{S}\setminus\{\text{sgn}(x)\}$ implies $s=-\text{sgn}(x)$ and, consequently, $\mathcal{B}(s,x)=-|x|$. Thus, $\mathcal{B}(\text{sgn}(x),x)>\mathcal{B}(s,x)$ for all $x\in\mathcal{D}$ and $s\in\mathcal{S}\setminus\{\text{sgn}(x)\}$. From Theorem 1, the bipolar Hopfield neural network always settles down at an equilibrium if $w_{ij}=w_{ji}$ and $w_{ii}\geq 0$.

This example highlights that the stability analysis of the traditional discrete-time Hopfield neural network is a particular case of the theory presented in this paper.

4.2 Complex-Valued Multistate Hopfield Neural Networks

As far as we know, although complex-valued Hopfield neural networks (CvHNNs) have been proposed in the late 1980s [49, 50], the most relevant contribution is the multistate model proposed by [25]. In fact, the multistate CvHNN of Jankowski et al. corroborated to the development of many other complex-valued Hopfield neural networks including the ones described on [32, 47, 43, 58, 31, 7, 24].

Mistakenly, Jankowski et al. stated that their multistate CvHNN, operating asynchronously, always settles down at an equilibrium state if the synaptic weight matrix satisfies the usual conditions: hermitian synaptic weights ($w_{ij}=\bar{w}_{ji}$) and non-negative self-connections ($w_{ii}\geq 0$). In 2014, Zhou and Zurada showed that the multistate CvHNN of Jankowski et al. may fail to yield a convergent sequence if $w_{ii}=0$ [68]. As pointed out by Zhou and Zurada, it turns out that the condition $w_{ii}=0$ is often used in applications of the multistate CvHNN. For instance, some design methods to implement an associative memory using multistate CvHNN, including the generalized projection rule [43], requires $w_{ii}=0$. Moreover, probably not aware of the subtle mistake by Jankowski et al., many multistate CvHNNs from the literature also fail to yield a convergent sequence of states under the usual conditions on the synaptic weights [7]. Based on Theorem 1, however, we provide below a solution to the stability analysis of multistate CvHNN under the usual conditions, including the case $w_{ii}=0,\forall i=1,\ldots,N$.

Consider the system of complex numbers $\mathbb{C}$ equipped with the natural conjugation given by (9). Note that the symmetric bilinear form $\mathcal{B}$ given by (10) satisfies

[TABLE]

where $z_{1}=|z_{1}|e^{\mathbf{i}\theta_{1}}$ and $z_{2}=|z_{2}|e^{\mathbf{i}\theta_{2}}$ are written in a polar representation such that $0\leq|\theta_{1}-\theta_{2}|\leq\pi$. In particular, $\mathcal{B}(z,z)=|z|^{2}\geq 0$ and, thus, $\mathbb{C}$ with the natural conjugation is a positive semi-definite real-part associative hypercomplex number system.

Motivated by [32, 31], let us define the complex-valued signum function as follows: Given a positive integer number $K>1$, define $\Delta\theta=\pi/K$. The integer $K$ and the angle $\Delta\theta$ are referred respectively to as the resolution factor and the phase-quanta. The complex-valued signum function $\text{csgn}:\mathcal{D}\to\mathcal{S}$ is defined by

[TABLE]

where

[TABLE]

denotes its domain and

[TABLE]

is the set of all possible states of a multistate complex-valued neuron. Note that csgn is not defined at a complex number $z$ such that $\arg(z)=(2k-1)\Delta\theta$, $k\in\{1,2,\ldots,K\}$. Therefore, from (20) with $f\equiv\text{csgn}$, a neuron of a multivalued CvHNN is updated at time $t$ if and only if $v_{i}(t)\neq 0$ and $\arg(v_{i}(t))\neq(2k-1)\Delta\theta$ for some $k\in\{1,2,\ldots,K\}$. This is the key issue to ensure that the sequences produced by a multistate CvMNN are all convergent. Precisely, in the following we show that csgn is a $\mathcal{B}$-projection function: Given a complex number $z=|z|e^{\mathbf{i}\theta}\in\mathcal{D}$, using the polar representation, let us write $\text{csgn}(z)=e^{\alpha\mathbf{i}}$ and $s=e^{\beta\mathbf{i}}$ for any $s\in\mathcal{S}\setminus\{\text{csgn}(z)\}$. From (26), we have

[TABLE]

By applying the cosine function, multiplying by $|z|$, and using (25), we obtain

[TABLE]

Concluding, from Theorem 1, the complex-valued multistate Hopfield neural network with $f\equiv\text{csgn}$ given by (26) yields a convergent sequence if $w_{ij}=\bar{w}_{ji}$ and $w_{ii}\geq 0$ for all $i,j\in\{1,\ldots,N\}$. Moreover, in contrast to the many models in the literature, a neuron is updated if only if its activation potential $v_{i}(t)\in\mathcal{D}$, that is, $v_{i}(t)\neq 0$ and $\arg(v_{i}(t))\neq(2k-1)\Delta\theta$ for some $k=1,\ldots,K$. At this point, note that the set $\mathbb{C}\setminus\mathcal{D}$ has Lebesgue measure zero. Thus, we almost always update a neuron using the csgn activation function. This remark justifies why the subtle mistake on the stability analysis of Jankowski et al. has been unaware for almost 18 years.

4.3 Tessarine-Valued Hopfield Neural Networks

As far as we know, Isokawa et al. were the first to investigate Hopfield neural networks on tessarines, also known as commutative quaternions [21]. Briefly, Isokawa and collaborators propose two tessarine-valued multistate discrete-time Hopfield neural networks based on polar representations of tessarines. Although we believe that both tessarine-valued Hopfield neural networks of Isokawa et al. can be analyzed using the theory presented previously in this paper, to simplify our discussion, in the following we only address a slight modification of the tessarine-valued multistate Hopfield neural network introduced recently by [36] and which is very similar to the second model proposed by [21].

Consider the hypercomplex number system $\mathbb{T}$ of tessarines equipped with the reverse-involution $\tau$ given by (13), i.e., $\tau(p)=p_{0}-p_{1}\mathbf{i}+p_{2}\mathbf{j}-p_{3}\mathbf{k}$ for all $p=p_{0}+p_{1}\mathbf{i}+p_{2}\mathbf{j}+p_{3}\mathbf{k}\in\mathbb{T}$. In this case, $\mathbb{T}$ is a positive semi-definite real-part associative hypercomplex number system and the symmetric bilinear form given by (10) satisfies $\mathcal{B}_{\mathbb{T}}(p,q)=p_{0}q_{0}+p_{1}q_{1}+p_{2}q_{2}+p_{3}q_{3}$ for all $p,q\in\mathbb{T}$. Since $\mathbf{k}=\mathbf{i}\mathbf{j}$, a tessarine can be written as the direct sum of two complex numbers $u_{p}=p_{0}+p_{1}\mathbf{i}$ and $v_{p}=p_{2}+p_{3}\mathbf{i}$ as follows:

[TABLE]

Moreover, because $\mathcal{B}_{\mathbb{C}}(z_{1},z_{2})=a_{1}a_{2}+b_{1}b_{2}$ for any complex-numbers $z_{1}=a_{1}+b_{1}\mathbf{i}$ and $z_{2}=a_{2}+b_{2}\mathbf{i}$, the symmetric bilinear form satisfies the following identity for any two tessarines $p=u_{p}+v_{p}\mathbf{j}$ and $q=u_{q}+v_{q}\mathbf{j}$:

[TABLE]

In analogy to the complex-valued multistate Hopfield neural network, given a positive integer number $K>1$ called resolution factor, define the phase-quanta by means of the equation $\Delta\theta=\pi/K$. Using (30), Kobayashi defines the set of all possible states of a tessarine-valued multistate neuron as follows:

[TABLE]

where $\mathcal{S}=\{1,e^{2\mathbf{i}\Delta\theta},e^{4\mathbf{i}\Delta\theta},\ldots,e^{2(K-1)\mathbf{i}\Delta\theta}\}\subset\mathbb{C}$. Also, in accordance with [36], let us define the tessarine-valued activation function $\text{tsgn}:\mathcal{D}_{\mathbb{T}}\to\mathcal{S}_{\mathbb{T}}$ by means of the equation

[TABLE]

where csgn denotes the complex-valued activation function given by (26) and

[TABLE]

For any $p=u_{p}+v_{p}\mathbf{j}\in\mathcal{D}_{\mathbb{T}}$ and $s=u_{s}+v_{s}\mathbf{j}\in\mathcal{S}_{\mathbb{T}}\setminus\{\text{tsgn}(p)\}$, from (29) and (31), we obtain

[TABLE]

Therefore, the tessarines $\mathbb{T}$ with the reverse-involution $\tau$ defined by (13) is a positive semi-definite real-part associative hypercomplex number system and $\text{tsgn}:\mathcal{D}_{\mathbb{T}}\to\mathcal{S}_{\mathbb{T}}$ is a $\mathcal{B}$-projection function. From Theorem 1, we conclude that the tessarine-valued multistate Hopfield neural network described by (20) yields a convergent sequence in an asynchronous update mode if the synaptic weights satisfy $w_{ij}=\tau(w_{ji})$ and $w_{ii}\geq 0$ for all $i,j=1,\ldots,N$.

Concluding, the theory presented in this paper provides a unified mathematical explanation for the stability analysis of many hypercomplex-valued neural networks such as the one detailed by [21] and [36].

4.4 Some Clifford-Valued Hopfield Neural Networks

As far as we know, Vallejo and Bayro-Corrochano provided the first account on discrete-time Hopfield neural networks on Clifford algebras [63]. Apart from the complex-valued HHNN, which is a real Clifford algebra of dimension equal to $2$, Hopfield neural networks on hyperbolic and dual domains have been investigated by [27, 29, 28, 35, 36]. Moreover, continuous-time Hopfield neural networks on Clifford algebras have been extensively investigated by [39, 40, 42]. In this subsection, we apply the theory presented in this paper to confirm the stability analysis of Hopfield neural networks on some real Clifford algebras. Precisely, we shall focus on Hopfield neural networks based on the so-called split-sign function and defined on real Clifford algebras of dimension 2.

A real Clifford algebra of dimension 2 corresponds to an associative hypercomplex number system with elements of the form $p=p_{0}+p_{1}\mathbf{i}$ where the square of the hypercomplex unit is a real number, that is, $\mathbf{i}^{2}\in\mathbb{R}$. As a consequence, a real Clifford algebra of dimension 2 is isomorphic to either the systems of complex numbers ($\mathbb{C}$), hyperbolic numbers ($\mathbb{U}$), or dual numbers ($\mathbb{D}$). We recall that the multiplication table of these three hypercomplex number systems are given by Table 1a), b), and c). Moreover, it is not hard to show that the systems of complex, hyperbolic, and dual numbers are all commutative algebras.

The Clifford conjugation corresponds to the natural conjugation on $\mathbb{C}$, $\mathbb{U}$, and $\mathbb{D}$. Apart from the natural conjugation, the identity mapping is also a reverse-involution on the complex, hyperbolic, and dual number systems. In view of this remark, it is convenient to introduce the notation

[TABLE]

where $\lambda\in\{-1,+1\}$. Note that $\tau_{\lambda}$ is the trivial reverse-involution if $\lambda=+1$ while, when $\lambda=-1$, we obtain the natural conjugation. In both cases, $\tau_{\lambda}$ is a reverse-involution on $\mathbb{C}$, $\mathbb{U}$, and $\mathbb{D}$. Furthermore, complex, hyperbolic, and dual number systems with $\tau_{\lambda}$ are all real-part associative hypercomplex number systems. The symmetric bilinear form given by (10) satisfies the following equation for all $p=p_{0}+p_{1}\mathbf{i}$ and $q=q_{0}+q_{1}\mathbf{i}$:

[TABLE]

where $\lambda\in\{-1,+1\}$ and $\mathbf{i}^{2}\in\{-1,0,+1\}$. In particular, we have $\mathcal{B}(p,p)\geq 0$ for all $p=p_{0}+p_{1}\mathbf{i}$ if and only if $\lambda\mathbf{i}^{2}\geq 0$. Thus, the system of complex numbers with the natural conjugation is a positive semi-definite real-part associative hypercomplex number system. Similarly, the system of hyperbolic numbers yields a positive semi-definite real-part associative hypercomplex number system with the identity mapping. Finally, the system of dual numbers is a positive semi-definite real-part associative hypercomplex number system with both the identity and the natural conjugation.

Now, consider the split-sign function $\text{sgn}:\mathcal{D}\to\mathcal{S}$ defined in a component-wise manner by means of the equation

[TABLE]

where

[TABLE]

and

[TABLE]

It can be easily verified that the inequality

[TABLE]

holds true for all $q=q_{0}+q_{1}\mathbf{i}\in\mathcal{D}$ and $s=s_{0}+s_{1}\mathbf{i}\in\mathcal{S}\setminus\{\text{sgn}(q)\}$ if and only if $\lambda\mathbf{i}^{2}\geq 0$. From Theorem 1, we conclude that a sequence produced by a Hopfield neural network with the split-sign function is convergent if $\lambda\mathbf{i}^{2}\geq 0$, $w_{ij}=\tau_{\lambda}(w_{ji})$, and $w_{ii}\geq 0$ for all $i,j=1,\dots,N$. In other words, a Hopfield neural network given by (20) with the split-sign function always yields a convergent sequence in the asynchronous update mode when:

– Complex numbers: The synaptic weights satisfy $w_{ij}=\bar{w}_{ji}$ and $w_{ii}\geq 0$ for all $i,j=1,\ldots,N$.

– Hyperbolic numbers: The synaptic weights satisfy, for all $i=1,\ldots,N$, the conditions $w_{ij}=w_{ji}$ and $w_{ii}\geq 0$. In a similar manner, the theory presented in this paper can be applied for the stability analysis of the two hyperbolic-valued neural networks proposed by [29, 36].

– Dual numbers: The synaptic weights satisfy $w_{ii}\geq 0$ and either $w_{ij}=\bar{w}_{ji}$ or $w_{ij}=w_{ji}$ holds true for all $i,j=1,\ldots,N$. Accordingly, [34] asserts that the identity $\text{Re}\left\{w_{ij}\right\}=\text{Re}\left\{w_{ji}\right\}$ ensures the stability of this recurrent neural network with $w_{ii}=0$ for all $i=1,\ldots,N$.

Another interesting result that emerges from the theory presented in this paper is obtained by considering the function $\overline{\text{sgn}}:\mathcal{D}\to\mathcal{S}$ defined by

[TABLE]

where $\mathcal{D}$ and $\mathcal{S}$ are given by (37). Note that $\overline{\text{sgn}}(p)=\text{sgn}(\bar{p})=\overline{\text{sgn}(p)}$. Therefore, the dynamic of the Hopfield neural networks on 2-dimensional real Clifford algebras with $\overline{\text{sgn}}$ corresponds to the recurrent neural networks obtained by considering sgn but either conjugating the activation potential or the output of the neuron. In contrast to Hopfield neural networks on 2-dimensional real Clifford algebras with the sgn function, the inequality

[TABLE]

holds true if and only if $\lambda\mathbf{i}^{2}\leq 0$. For example, (39) holds if $\mathbf{i}^{2}=-1$ and $\lambda=1$. In other words, $\overline{\text{sgn}}$ is a $\mathcal{B}$-projection function on the complex numbers equipped with the trivial reverse-involution. Although $\mathbb{C}$ with the identity mapping is a real-part associative hypercomplex number system, it is not positive semi-definite. Thus, from Theorem 1, we conclude that a complex-valued Hopfield neural network with the activation function (36) and asynchronous update always comes to rest at an equilibrium if the synaptic weights satisfy $w_{ij}=w_{ji}$ and $w_{ii}=0$. Accordingly, we have just derived the stability conditions for the symmetric complex-valued Hopfield neural network proposed by [32] and further discussed using the split-sign activation function by [33]. Similarly, we conclude that an HHNN given by (20) with the $\overline{\text{sgn}}$ activation function always settle down at a stationary state in the asynchronous update mode if:

– Complex numbers: The synaptic weights satisfy $w_{ii}=0$ and $w_{ij}={w}_{ji}$ for all $i,j=1,\ldots,N$.

– Hyperbolic numbers: The synaptic weights satisfy $w_{ii}=0$ and $w_{ij}=\bar{w}_{ji}$ for all $i,j=1,\ldots,N$.

– Dual numbers: The synaptic weights satisfy $w_{ii}\geq 0$ and either $w_{ij}=\bar{w}_{ji}$ or $w_{ij}=w_{ji}$ holds true for all $i,j=1,\ldots,N$.

Let us conclude this subsection with two simple illustrative examples.

Example 4**.**

Consider the four-state Hopfield neural network with $N=2$ neurons on the three real Clifford algebras of dimension 2 and synaptic weights given by

[TABLE]

Note that this Hopfield neural network has symmetric weights and no self-feedback. Thus, the complex-valued neural network with $\overline{\text{sgn}}$ activation function, the hyperbolic-valued neural network with sgn activation function, and both dual-numbered neural networks always come to rest at an equilibrium. In contrast, the complex-valued neural network with sgn and the hyperbolic-valued neural network with $\overline{\text{sgn}}$ may fail to settle down at a stationary state. Indeed, Figure 2 shows the directed graphs obtained from the six Hopfield neural networks on the real Clifford algebras of dimension 2. In these directed graphs, a node corresponds to a state of the neural network while an edge from node $i$ to node $j$ means that we can obtain the $j$th state from the $i$th state by updating a single neuron. For illustractive purposes, we depicted with red edges all the possible trajectories obtained from the initial state $\mathbf{x}(0)=[-1-\mathbf{i},1+\mathbf{i}]$, which corresponds to the state number four in the directed graphs. Note from Figure 2 that there is a one-to-one correspondence between the a) complex-valued HNN with $\overline{\text{sgn}}$ and b) the hyperbolic-valued HNN with the sgn. Although these two HHNN models settle down at different equilibrium states, they exhibit very similar dynamics. Similarly, there is also a one-to-one correspondence between e) the complex-valued HNN with sgn and f) the hyperbolic-valued HNN with the $\overline{\text{sgn}}$. In particular, these two models exhibit limit cycles which prevent the HNNs to settle down at a stationary state. Finally, observe that the dynamic of the dual-numbered HNN with sgn activation function is similar to the dynamic of the dual-numbered HNN with $\overline{\text{sgn}}$ – both models have the same number of equilibrium points. Concluding, apart from an application of the theory presented in this paper for stability analysis of several models from the literature, this example reveals an interesting relationship between the Hopfield neural networks on real Clifford algebras of dimension 2 which requires further study.

Example 5**.**

The previous example addressed the dynamic of HHNNs on real Clifford algebras of dimension 2. Let us now compare these HHNNs with the split-sign activation function with their corresponding real-valued versions. Precisely, a complex number $a_{1}+b_{1}\mathbf{i}$, a hyperbolic number $a_{2}+b_{2}\mathbf{i}$, and a dual-number $a_{3}+b_{3}\mathbf{i}$ can be identified, respectively, with the following $2\times 2$-matrices:

[TABLE]

As a consequence, the complex-valued, hyperbolic-valued, and dual-numbered Hopfield neural networks with the split-sign activation function of Example 4 can be identified with real-valued bipolar Hopfield neural networks with the split-sign activation function and four neurons $(N=4$) whose synaptic weight matrices are given respectively by

[TABLE]

Note that only $W^{h}$ is symmetric with non-negative diagonal. Hence, the real-valued neural network derived from the symmetric hyperbolic-valued neural network always settles down at an equilibrium point. From the classical result for bipolar Hopfield neural network [18], however, nothing can be said about the other two real-valued neural networks. The directed graphs depicted on Figure 3 illustrate the dynamic of the three real-valued Hopfield neural networks. In analogy to Figure 2, all possible trajectories obtained by starting the neural network at $\mathbf{x}(0)=[-1-\mathbf{i},1+\mathbf{i}]$, which corresponds to the state number four, have been depicted using red edges. Note that the real-valued neural network obtained from the complex-valued Hopfield neural network also exhibits limit cycles. Precisely, in contrast to the complex-valued neural network, its corresponding real-valued Hopfield neural network may exhibit a chaotic behavior because any state can be reached from any initial state. Although the hypercomplex-valued Hopfield neural network and its corresponding real-valued neural network have the same stationary states, they exhibit different dynamics. Indeed, the hyperbolic-valued Hopfield neural network settles at an equilibrium with at most one single neuron update. The real-valued Hopfield neural network, however, may require two updates to reach a stationary state. Furthermore, starting at state number four, the corresponding real-valued neural network can reach any of the four possible stationary states (follow the red edges in the graph in Figure 3b)). In contrast, starting at the same initial state, the hyperbolic-valued HNN can settle only on two of the four stationary states (see the red edges in the graph in Figure 2b)). In particular, the hyperbolic neural network has 50% of chance to settle at the stationary state number 1 while the real-valued as only 25% of chance to settle at the same stationary state. Putting this remark in a practical context, if neural networks are designed to implement associative memories [18, 14], the real-valued Hopfield neural network is more likely to settle at a spurious memory than the hyperbolic-valued neural network. Finally, the dual-numbered Hopfield neural network and its corresponding real-valued neural network also have the same stationary states. The real-valued Hopfiled neural network, however, usually require more updates to settle at an equilibrium than the dual-numbered model. Concluding, this example suggests that hypercomplex-valued Hopfield neural networks may be simpler (e.g. in terms of the complexity of the resulting directed graphs) than their corresponding real-valued models. In particular, in this example the hypercomplex-valued models usually required fewer updates to settle down at an equilibrium state than their corresponding real-valued neural networks.

Apart from the models in the literature, the following subsection reveals that the theory presented in this paper can be used to study the dynamic of novel HHNN models.

4.5 HHNNs on Cayley-Dickson Algebras

In analogy to the multistate CvHNN, a continuous-valued CvHNN is obtained by considering in (20) the activation function defined by $\sigma(z)=z/|z|$ for all $z\neq 0$. The complex-valued discrete-time Hopfield neural network obtained by considering $\sigma$ as the activation function has been investigated by [50]. More recently, the continuous-valued CvHNN has been extended to quaternions independently by [60] and [30]. We also used this kind of activation function to introduce a discrete-time continuous-valued octonionic Hopfield neural network [6]. At this point, we would like to recall that Kuroe and Iima have investigated the stability of continuous-time octonionic Hopfield neural networks [41]. Their study motivated us to investigate the discrete-time models and inspired us to introduce the theory presented in this paper. Interestingly, the complex, quaternionic, and octonionic continuous-valued Hopfield neural networks, operating asynchronously, yield a convergent sequence of states if the synaptic weights satisfy the usual conditions: $w_{ij}=\bar{w}_{ji}$ and $w_{ii}\geq 0$ for all $i,j=1,\ldots,N$. Since the systems of complex numbers, quaternions, and octonions are all instances of Cayley-Dickson algebras, let us apply the theory developed in this paper for the analysis of the stability of HHNN models defined on an arbitrary Cayley-Dickson algebra $A_{k}$.

As pointed out previously, a Cayley-Dickson algebra $A_{k}$ whose conjugation and product are defined recursively by (15) and (16) with $A_{0}=\mathbb{R}$ is a positive semi-definite real-part associative hypercomplex number system. Although the multiplication may fail to have some desirable algebraic properties such as commutativity and associativity, the Cayley-Dickson algebra $A_{k}$ enjoys some properties from Euclidean geometry. Precisely, from (19), the symmetric bilinear form $\mathcal{B}:A_{k}\times A_{k}\to\mathbb{R}$ corresponds to the usual inner product between $p={p}_{0}+{p}_{1}\mathbf{i}_{1}+\ldots+{p}_{n}\mathbf{i}_{n}\equiv(p_{0},p_{1},\ldots,p_{n})$ and $q={q}_{0}+{q}_{1}\mathbf{i}_{1}+\ldots+{q}_{n}\mathbf{i}_{n}\equiv(q_{0},q_{1},\ldots,q_{n})$, where $n=2^{k}-1$. Moreover, the absolute value of a hypercomplex number $p\equiv(p_{0},p_{1},\ldots,p_{n})\in A_{k}$ can be defined by

[TABLE]

which corresponds to the Euclidean norm. From the Cauchy-Schwarz inequality, we have $\mathcal{B}(p,q)\leq|p||q|$, with equality if and only if $q=\alpha p$ for some real $\alpha>0$.

In analogy to the complex-valued, quanternionic, and octonionic continuous-valued HHNNs, let $\sigma:\mathcal{D}\to\mathbb{S}$ be the activation function given by

[TABLE]

where $\mathbb{S}=\{p\in A_{k}:|p|=1\}$. The following shows that $\sigma$ is a $\mathcal{B}$-projection function. Consider $p\in\mathcal{D}$ and $s\in\mathcal{S}\setminus\{\sigma(p)\}$. Since $s\neq\alpha p$, for any $\alpha>0$, we obtain

[TABLE]

The inequality is a consequence of the Cauchy-Schwarz inequality and the fact that $p$ and $s$ are not parallel vectors. Thus, $\mathcal{B}(\sigma(p),p)>\mathcal{B}(s,p)$ for all $p\in\mathcal{D}$ and $s\in\mathbb{S}\setminus\left\{\sigma(p)\right\}$. Since $A_{k}$ is a positive semi-definite real-part associative hypercomplex number system, from Theorem 1, the sequences generated by (20) with $f\equiv\sigma$ are all convergent if the synaptic weights satisfy the usual conditions $w_{ij}=\bar{w}_{ji}$ and $w_{ii}\geq 0$.

In analogy to the continuous-valued Hopfield neural networks, it is rather straightforward to extend the real-valued, complex-valued, and quaternion-valued Hopfield neural networks with split-sign activation function to Cayley-Dickson algebras. Precisely, consider a Cayley-Dickson algebra $A_{k}$ and let the set of states of a neuron be

[TABLE]

where $n=2^{k}-1$. Also, consider the split-sign activation function $\text{sgn}:\mathcal{D}\to\mathcal{S}$ defined in a component-wise manner as follows for all $q\in\mathcal{D}$, where $\mathcal{D}=\{q\in A_{k}:q_{0}q_{1}\cdots q_{n}\neq 0\}$:

[TABLE]

Geometrically, the split-sign function projects its argument onto a vertex of a hypercube of a $2^{k}$-dimensional space. Since the symmetric bilinear form $\mathcal{B}:A_{k}\times A_{k}\to\mathbb{R}$ corresponds to the usual inner product, the following holds true for any $q\in\mathcal{D}$ and $s\in\mathcal{S}\setminus\{\text{sgn}(q)\}$:

[TABLE]

The inequality (42) follows because there exists an index $\nu\in\{0,1,\ldots,n\}$ such that $s_{\nu}=-1$ if $q_{\nu}>0$ and $s_{\nu}=+1$ if $q_{\nu}<0$. Equivalently, $s_{\nu}q_{\nu}=-|q_{\nu}|$ for some $\nu$. From (43), we conclude that the split-sign is also a $\mathcal{B}$-projection function on a Cayley-Dickson algebra $A_{k}$. Furthermore, from Theorem 1, the sequences generated by (20) with $f\equiv\text{sgn}$ are all convergent if the synaptic weights satisfy the usual conditions $w_{ij}=\bar{w}_{ji}$ and $w_{ii}\geq 0$.

Example 6**.**

Let us illustrate the dynamic of an octonion-valued (Cayley-Dickson algebra $A_{3}$) Hopfield neural network with the split-sign activation function and compare it with its corresponding real-valued bipolar Hopfield neural network. Precisely, we synthesized an octonion-valued Hopfield neural network with $N=100$ neurons and weights defined as follows for $i=1,\ldots,N$ and $j=i+1,\ldots,N$:

[TABLE]

Here, $\mathtt{randn}$ yields a real number sampled from a normal probability distribution with mean 0 and standard deviation 1. The corresponding real-valued neural network is obtained by considering a bipolar Hopfield neural network with $8N$ neurons and real-valued synaptic weight matrix $M$ such that $M\phi(\mathbf{x})=\phi(W\mathbf{x})$, where $\phi:\mathbb{O}^{N}\to\mathbb{R}^{8N}$ is the bijection defined as follows for any $\mathbf{x}=[({x_{1}}_{0}+{x_{1}}_{1}\mathbf{i}_{1}+\ldots+{x_{1}}_{7}\mathbf{i}_{7}),\ldots,({x_{N}}_{0}+{x_{N}}_{1}\mathbf{i}_{1}+\ldots+{x_{N}}_{7}\mathbf{i}_{7})]\in\mathbb{O}^{N}$:

[TABLE]

In words, the multiplication of the real-valued synaptic weight matrix $M$ by the real-valued vector $\phi(\mathbf{x})$ obtained by concatenating all the entries of the octonion-valued vector $\mathbf{x}$ corresponds to the real-valued vector obtained by concatenating the entries of the product between the octonion-valued matrix $W$ and $\mathbf{x}$ [59]. Furthermore, the hypercomplex-valued neural network has been initialized at an octonion-valued vector $\mathbf{x}(0)\in\mathcal{S}^{N}$ whose components $x_{i}(0)={x_{i}}_{0}(0)+{x_{i}}_{1}(0)\mathbf{i}_{1}+\ldots+{x_{i}}_{7}(0)\mathbf{i}_{7}$ are uniformly distributed in $\mathcal{S}$, that is, $\mbox{Pr}[{x_{i}}_{\mu}(0)=+1]=0.5$ for all $i=1,\ldots,N$ and $\mu=0,\ldots,7$. The real-valued Hopfield neural network has been initialized with the $8N$-dimensional bipolar vector $\phi(\mathbf{x}(0))$ obtained by concatenating all the entries of the octonion-valued input. The evolution of the energy given by (22) for both octonion-valued and real-valued Hopfield neural network from one outcome of this experiment is shown in Fig. 4. Both neural networks evolved until they reach a stationary state. Note that the octonion-valued as well as the real-valued model settled at a stationary state whose energy value is close to $-16000$. Nevertheless, the octonion-valued Hopfield neural network reached a steady state with much less updates than its corresponding real-valued neural network. Intuitively, the fast convergence of the octonion-valued neural network follows because one hypercomplex-valued neuron can be viewed as a group of 8 real-valued neurons. Furthermore, we may conjecture that hypercomplex-valued Hopfield neural networks overcome real-valued models when a certain entity can be described by a hypercomplex number or a group of real-valued neurons. In the future, we intend to develop efficient learning strategies for hypercomplex-valued Hopfield neural networks and further compare their performance with real-valued models.

Concluding, the stability analysis of the complex-valued [50], quaternionic [60, 62, 30], and octonionic-valued [6] Hopfield neural networks can be derived as a particular case of the theory presented in this paper. More generally, this theory can be applied in a straightforward manner to the broad class of continuous-valued and split-sign Hopfield neural networks defined on a Cayley-Dickson algebra $A_{k}$.

5 Concluding Remarks

In this paper, we addressed the stability of a broad class of discrete-time hypercomplex-valued Hopfield-type neural networks (HHNNs). To this end, we introduced new hypercomplex number systems called real-part associative hypercomplex number systems. Real-part associative hypercomplex number systems provide an appropriate mathematical background for the development of HHNNs and include, as particular instances, Cayley-Dickson algebras and real Clifford algebras.

Apart from the new hypercomplex number systems, in this paper we also introduced a broad family of hypercomplex-valued functions, referred to as $\mathcal{B}$-projection functions. The stability of an HHNN with a $\mathcal{B}$-projection function is ensured by means of Theorem 1 under mild conditions on the synaptic weights. It should be emphasized that the results presented in this paper extend several results published in the literature on the stability analysis of discrete-time Hopfield-type neural networks since the early 1980s. Moreover, it can be applied for the development of many new HHNN models. Indeed, we used the theory presented in this paper to introduce a broad class of HHNNs on Cayley-Dickson algebras.

Finally, we would like to recall that the real-valued Hopfield neural network has been applied in control [12, 56], computer vision and image processing [65, 45], classification [51, 67], optimization [19, 53, 44], and to implement associative memories [18, 14]. Likewise, using the background theory presented in this paper, we believe that HHNNs can be applied in control, computer vision and image processing, to solve optimization problems as well as to implement associative memories designed for the storage and recall of multidimensional data. Indeed, complex-valued and quaternion-valued Hopfield neural networks have been effectively applied to implement associative memories, for example in [25, 22, 24, 58]. In particular, we intend to develop efficient learning rules for HHNNs in the future.

Appendix – Proof of Theorem 1

First of all, since $\mathcal{S}$ is compact, the Cartesian product $\mathcal{S}^{N}$ is also compact. Furthermore, the function $E$ given by (22) (or, equivalently by (23)) is real-valued, by definition. Also, it is continuous because $\mathcal{B}$ is a symmetric bilinear form on a finite dimensional vector space. Now, a well-known result from calculus for continuous function ensures that the continuous image of a compact set is compact (see Corollary 2.5-7 of [37] or Theorem 4.25 of [3]). Therefore, $E:\mathbb{S}^{N}\to\mathbb{R}$ attains its maximum and minimum values at some points of $\mathcal{S}^{N}$. In other words, $E$ is bounded. Let us now show that $E$ is strictly decreasing along any non-stationary trajectory.

Aiming to simplify notation, let $\mathbf{x}\equiv\mathbf{x}(t)$ and $\mathbf{x}^{\prime}\equiv\mathbf{x}(t+\Delta t)$ for some $t\geq 0$. Since we are considering an asynchronous update mode, let us suppose that only the $\mu$th neuron changed its state at iteration $t$. In other words, we assume that $x_{j}^{\prime}=x_{j}$ for all $j\neq\mu$ and $x_{\mu}^{\prime}\neq x_{\mu}$. In this case, the function $E$ evaluated at $\mathbf{x}$ and $\mathbf{x}^{\prime}$ satisfy:

[TABLE]

and, since $x_{j}^{\prime}=x_{j}$ for all $j\neq\mu$, we have

[TABLE]

Hence, using the linearity of $\mathcal{B}$, the variation of the energy from time $t$ to $t+\Delta t$ is

[TABLE]

Replacing $i$ by $j$ in the second sum, using the identity (12), the symmetry of $\mathcal{B}$, and recalling that $\tau(w_{j\mu})=w_{\mu j}$, we obtain

[TABLE]

From the linearity and symmetry of $\mathcal{B}$, using (12) again, and recalling that the activation potential of the $\mu$th neuron at iteration $t$ can be expressed as

[TABLE]

we have

[TABLE]

Now, since we are assuming $f(v_{\mu})=x_{\mu}^{\prime}\neq x_{\mu}$, we must have $v_{\mu}\in\mathcal{D}$. Moreover, since $f$ is a $\mathcal{B}$-projection function and $x_{\mu}\neq f(v_{\mu})$, we obtain

[TABLE]

As a consequence, we have

[TABLE]

On the one hand, if $w_{ii}=0$ for all $i=1,\ldots,N$, then

[TABLE]

which concludes the proof in the case (a). On the other hand, if $\mathbb{H}$ is a positive semi-definite real-part associative hypercomplex number system, then $\mathcal{B}(x_{\mu}^{\prime}-x_{\mu},x_{\mu}^{\prime}-x_{\mu})\geq 0$. Also, if $w_{\mu\mu}$ is a non-negative real number, then

[TABLE]

which concludes the proof in the case (b).

Acknowledgment

This work was supported in part by CNPq under grant no. 310118/2017-4 and FAPESP under grant no. 2019/02278-2.