On Reproducing Kernel Banach Spaces: Generic Definitions and Unified Framework of Constructions

TL;DR

This paper introduces a generic, unified framework for constructing reproducing kernel Banach spaces (RKBS), clarifies their relations, and develops corresponding machine learning theorems, including a new class of Orlicz RKBSs.

Contribution

It proposes a construction-independent, generic definition of RKBS and a unifying framework that encompasses existing RKBS constructions and introduces Orlicz RKBSs.

Findings

Unified framework for RKBS constructions using bilinear forms and feature maps.

Development of representer theorems applicable across the unified RKBS framework.

Introduction of a new class of Orlicz RKBSs.

Abstract

Recently, there has been emerging interest in constructing reproducing kernel Banach spaces (RKBS) for applied and theoretical purposes such as machine learning, sampling reconstruction, sparse approximation and functional analysis. Existing constructions include the reflexive RKBS via a bilinear form, the semi-inner-product RKBS, the RKBS with $\ell^1$ norm, the $p$-norm RKBS via generalized Mercer kernels, etc. The definitions of RKBS and the associated reproducing kernel in those references are dependent on the construction. Moreover, relations among those constructions are unclear. We explore a generic definition of RKBS and the reproducing kernel for RKBS that is independent of construction. Furthermore, we propose a framework of constructing RKBSs that unifies existing constructions mentioned above via a continuous bilinear form and a pair of feature maps. A new class of Orlicz RKBSs is proposed. Finally, we develop representer theorems for machine learning in RKBSs constructed in our framework, which also unifies representer theorems in existing RKBSs.

Equations

$$|{\delta }_x(f)|=|f(x)|\le C_x\Vert f{\Vert }_{\mathcal{B}}\text{mbox}forallf\in \mathcal{B}.$$

$$|⟨f,g{⟩}_{V_1\times V_2}|\le C\Vert f{\Vert }_{V_1}\Vert g{\Vert }_{V_2}\text{mbox}forallf\in V_1,g\in V_2.$$

$$f(x)=⟨f,K(x,\cdot ){⟩}_{{\mathcal{B}}_1\times {\mathcal{B}}_2}\text{mbox}forallx\in {\Omega }_1\text{mbox}andallf\in {\mathcal{B}}_1,$$

$$g(y)=⟨K(\cdot ,y),g{⟩}_{{\mathcal{B}}_1\times {\mathcal{B}}_2}\text{mbox}forally\in {\Omega }_2\text{mbox}andallg\in {\mathcal{B}}_2,$$

$$⟨a,v{⟩}_{{\mathcal{W}}_1\times {\mathcal{W}}_2}=0\text{mbox}foralla\in A$$

$${\Phi }_1:{\Omega }_1\to {\mathcal{W}}_1,{\Phi }_2:{\Omega }_2\to {\mathcal{W}}_2$$

$$\mathrm{span}{\Phi }_1({\Omega }_1)\text{mbox}isdensein{\mathcal{W}}_1,\mathrm{span}{\Phi }_2({\Omega }_2)\text{mbox}isdensein{\mathcal{W}}_2.$$

$${\cal B}_{1}:=\Big{\{}f_{v}(x):=\langle\Phi_{1}(x),v\rangle_{{\cal W}_{1}\times{\cal W}_{2}}:v\in{\cal W}_{2},x\in\Omega_{1}\Big{\}}$$

$$\Vert f_v{\Vert }_{{\mathcal{B}}_1}:=\Vert v{\Vert }_{{\mathcal{W}}_2}$$

$${\cal B}_{2}:=\Big{\{}g_{u}(y):=\langle u,\Phi_{2}(y)\rangle_{{\cal W}_{1}\times{\cal W}_{2}}:u\in{\cal W}_{1},y\in\Omega_{2}\Big{\}}$$

$$\Vert g_u{\Vert }_{{\mathcal{B}}_2}:=\Vert u{\Vert }_{{\mathcal{W}}_1}.$$

$$⟨u,v{⟩}_{{\mathcal{W}}_1\times {\mathcal{W}}_2}=⟨u,v{⟩}_{{\mathcal{W}}_1}:=v(u),u\in {\mathcal{W}}_1,v\in {\mathcal{W}}_2.$$

$${\mathcal{W}}_1=\overline{\mathrm{span}}{\Phi }_1({\Omega }_1),{\mathcal{W}}_1^{\ast }=\overline{\mathrm{span}}{\Phi }_2({\Omega }_2)$$

$${\mathcal{W}}_1=\overline{\mathrm{span}}{\Phi }_1({\Omega }_1),\text{mbox}and\mathrm{span}{\Phi }_2({\Omega }_2)\text{mbox}isdensein{\mathcal{W}}_1^{\ast }\text{mbox}{undertheweak}^{\ast }\text{mbox}topology.$$

$$⟨f_v,g_u{⟩}_{{\mathcal{B}}_1\times {\mathcal{B}}_2}:=⟨u,v{⟩}_{{\mathcal{W}}_1\times {\mathcal{W}}_2}\text{mbox}forall{f}_v\in {\mathcal{B}}_1\text{mbox}andall{g}_u\in {\mathcal{B}}_2,$$

$$K(x,y):=⟨{\Phi }_1(x),{\Phi }_2(y){⟩}_{{\mathcal{W}}_1\times {\mathcal{W}}_2},x\in {\Omega }_1,y\in {\Omega }_2,$$

$$|⟨u,v{⟩}_{{\mathcal{W}}_1\times {\mathcal{W}}_2}|\le C\Vert u{\Vert }_{{\mathcal{W}}_1}\Vert v{\Vert }_{{\mathcal{W}}_2}\text{mbox}forallu\in {\mathcal{W}}_1\text{mbox}andallv\in {\mathcal{W}}_2.$$

$$|f_v(x)|=|⟨{\Phi }_1(x),v{⟩}_{{\mathcal{W}}_1\times {\mathcal{W}}_2}|\le C\Vert {\Phi }_1(x){\Vert }_{{\mathcal{W}}_1}\Vert v{\Vert }_{{\mathcal{W}}_2}=C\Vert {\Phi }_1(x){\Vert }_{{\mathcal{W}}_1}\Vert f_v{\Vert }_{{\mathcal{B}}_1}.$$

$$|g_u(y)|=|⟨u,{\Phi }_2(y){⟩}_{{\mathcal{W}}_1\times {\mathcal{W}}_2}|\le C\Vert {\Phi }_2(y){\Vert }_{{\mathcal{W}}_2}\Vert u{\Vert }_{{\mathcal{W}}_1}=C\Vert {\Phi }_2(y){\Vert }_{{\mathcal{W}}_2}\Vert g_u{\Vert }_{{\mathcal{B}}_2}.$$

$$|⟨f_v,g_u{⟩}_{{\mathcal{B}}_1\times {\mathcal{B}}_2}|=|⟨u,v{⟩}_{{\mathcal{W}}_1\times {\mathcal{W}}_2}|\le C\Vert u{\Vert }_{{\mathcal{W}}_1}\Vert v{\Vert }_{{\mathcal{W}}_2}=C\Vert f_v{\Vert }_{{\mathcal{B}}_1}\Vert g_u{\Vert }_{{\mathcal{B}}_2}.$$

$$K(x,\cdot )=g_{{\Phi }_1(x)}\in {\mathcal{B}}_2,f_v(x)=⟨{\Phi }_1(x),v{⟩}_{{\mathcal{W}}_1\times {\mathcal{W}}_2}=⟨f_v,g_{{\Phi }_1(x)}{⟩}_{{\mathcal{B}}_1\times {\mathcal{B}}_2}=⟨f_v,K(x,\cdot ){⟩}_{{\mathcal{B}}_1\times {\mathcal{B}}_2}$$

$$K(\cdot ,y)=f_{{\Phi }_2(y)}\in {\mathcal{B}}_1,g_u(y)=⟨u,{\Phi }_2(y){⟩}_{{\mathcal{W}}_1\times {\mathcal{W}}_2}=⟨f_{{\Phi }_2(y)},g_u{⟩}_{{\mathcal{B}}_1\times {\mathcal{B}}_2}=⟨K(\cdot ,y),g_u{⟩}_{{\mathcal{B}}_1\times {\mathcal{B}}_2}$$

$${\mathcal{W}}_1=\overline{\mathrm{span}}\{{\delta }_x:x\in X\}\text{mbox}in{\mathcal{B}}^{\ast },{\Omega }_1=X,\text{mbox}and{\Phi }_1(x)={\delta }_x.$$

$${\mathcal{W}}_2=\mathcal{B},{\Omega }_2=\mathbb{N},\text{mbox}and{\Phi }_2(n)=f_n(x).$$

$$K(x,n)=⟨{\delta }_x,f_n⟩=f_n(x),x\in {\Omega }_1,n\in \mathbb{N}$$

$$\psi (t):={\phi }^{-1}(t),\Phi (t):={\int }_0^t\phi (s)ds,\Psi (t):={\int }_0^t\psi (s)ds,t\in {\mathbb{R}}_{+}.$$

$$xy\le \Phi (x)+\Psi (y),x,y\in {\mathbb{R}}_{+},$$

$${\int }_{\Omega }\Phi (\alpha |f|)d\mu <+\infty \text{mbox}forsome\alpha >0.$$

$$\|f\|_{\Phi}:=\inf\Big{\{}\alpha>0:\int_{\Omega}\Phi\Big{(}\frac{|f|}{\alpha}\Big{)}d\mu\leq\Phi(1)\Big{\}},\ \ f\in{\cal L}^{\Phi}$$

$$|f|_{\Phi}:=\sup\Big{\{}\int_{\Omega}|fg|d\mu:g\in{\cal L}^{\Psi},\ \|g\|_{\Psi}\leq 1\Big{\}},\ \ f\in{\cal L}^{\Phi}.$$

Full text

Keywords: Reproducing kernel Banach spaces, feature maps, reproducing kernels, machine learning, the representer theorem

1 Introduction

In this paper, we aim at the construction of reproducing kernel Banach spaces (RKBSs), which serve as a generalization of reproducing kernel Hilbert spaces (RKHSs) [33]. The notion of RKBS was originally introduced in machine learning in 2009 [46]. Since then, various RKBSs [13, 16, 37, 38, 43, 45, 46, 48] have been constructed for different applied and theoretical purposes. They are used in a wide variety of fields such as machine learning [43, 45, 46, 48, 49], sampling reconstruction [14, 15, 18, 27, 28], sparse approximation [37, 38, 43, 45], and functional analysis [9, 35, 47]. The definitions of existing RKBSs and the associated reproducing kernels in the literature are dependent on the construction. To address this issue and to further promote research in the subject, it is helpful to understand the essence of RKBS and to propose a framework of constructing RKBSs that unifies existing constructions.

RKHSs are Hilbert spaces of functions on which point evaluation functionals are continuous [33]. In machine learning, RKHSs have been viewed as ideal spaces for kernel-based learning algorithms. Thanks to the existence of an inner product, Hilbert spaces are well-understood in functional analysis. Many applications require that the sampling process to be stable. In other words, the point evaluations should be bounded. Most importantly, an RKHS has a reproducing kernel, which measures similarity between input data and gives birth to the “kernel trick” in machine learning that significantly saves computations. Successful and important machine learning methods based on RKHSs include support vector machines and the regularization networks [33, 36].

There are many reasons that justify the need of RKBSs. We mention three here. First of all, Banach spaces possess richer geometrical structures and norms. It is well-known that any two Hilbert spaces over $\mathbb{C}$ of the same dimension are isometrically isomorphic. By contrast, for $1\leq p\neq q\leq+\infty$, $L^{p}([0,1])$ and $L^{q}([0,1])$ are not isomorphic to each other (see, [12], page 180). Secondly, kernel functions play the role of measuring similarity of inputs in machine learning. They are defined by inner product through a feature map and therefore are inherently symmetric. In some applications such as psychology [51], asymmetric kernels are desired, which can only be obtained via Banach spaces. Thirdly, machine learning schemes in Banach spaces have received considerable attention recently [1, 10, 25, 26, 37, 38, 43, 45, 46, 48, 49]. Many important problems such as $p$-norm coefficient-based regularization [34, 38, 40, 42], large-margin classification [10, 45, 46], lasso in statistics [39] and compressed sensing [4] had better be studied in Banach spaces. It suggests the need of extending Hilbert space type arguments to Banach spaces.

Under these considerations, different definitions and constructions of RKBSs have been proposed in the literature. In 2009, Zhang et al. [46] proposed the new concept of RKBS for machine learning. Reflexive RKBSs were constructed by the bilinear form between a reflexive Banach space and its dual. The semi-inner-product RKBS was also studied in [46] and [47, 48] based on the tool of semi-inner products [17, 22]. For the multi-task learning, Zhang et al. [49] developed the notion of semi-inner-product vector-valued RKBS. In 2013, Song et al. [38] constructed a class of RKBSs with the $\ell^{1}$ norm via admissible kernels targeting at sparse learning. In those spaces, the representer theorem for regularized learning schemes is satisfied. In 2014, Georgiev et al. [16] constructed a class of RKBSs with the $p$-norm ($1\leq p\leq+\infty$) without the representer theorem. In 2015, Fasshauer et al. [13] constructed a class of RKBSs with positive definite functions. More recently, the $p$-norm ($1\leq p\leq+\infty$) RKBSs were systematically developed by Xu and Ye in [43] via generalized Mercer kernels.

The definitions of RKBSs and the associated reproducing kernels in the above references are dependent on the construction. Moreover, the relation among those constructions is unclear. Limitations also persist. For instance, the Banach space $C([0,1])$ of all continuous functions on the interval $[0,1]$ does not satisfy those definitions. However, $C([0,1])$ should be an RKBS as point evaluations are clearly continuous in the space. We hope to propose a framework of constructing RKBSs that unifies existing constructions, and leads further to new constructions that accept $C([0,1])$ as a particular example. The main purpose of this paper is two-fold:

(i)

to give a generic definition of RKBS that naturally generalizes the classical RKHS and is independent of construction;

(ii)

to propose a unified framework of constructing RKBSs that covers all existing constructions in the literature and also leads to new RKBSs.

The outline of the paper is as follows. In Section 2, we first give a generic definition of RKBSs, and then propose a novel framework of constructing RKBSs via a pair of feature maps. Furthermore, we are able to construct a new class of Orlicz RKBSs. In Section 3, we justify that our framework does unify existing RKBSs in the literature. In particular, $C([0,1])$ is included. In the last section, we develop a representer theorem for regularization networks in RKBSs constructed in our framework.

2 Generic definitions and constructions

In this section, we first present a generic definition of RKBS and the reproducing kernel for RKBS. Then, we propose a novel framework of constructing RKBSs via a pair of feature maps. Finally, we construct new RKBSs based on Orlicz spaces.

2.1 Generic definitions of RKBSs

We describe our generic definition of RKBSs as follows.

Definition 2.1

(Reproducing Kernel Banach Spaces (RKBS)) A reproducing kernel Banach space ${\cal B}$ on a prescribed nonempty set $X$ is a Banach space of certain functions on $X$ such that every point evaluation functional $\delta_{x}$, $x\in X$ on ${\cal B}$ is continuous, that is, there exists a positive constant $C_{x}$ such that

[TABLE]

Note that a normed vector space $V$ on $X$ is called a Banach space of functions if it is a Banach space whose elements are functions on $X$ and for each $f\in V$, $\|f\|_{V}=0$ if and only if $f$, as a function, vanishes everywhere on $X$. By definition, $L^{p}([0,1])$, $1\leq p\leq+\infty$, is not a Banach space of functions as it consists of equivalent classes of functions with respect to the Lebesgue measure.

Definition 2.1 naturally generalizes the classical definition of RKHS. We should point out that such a definition was implicitly mentioned in some papers on sampling theorems [5, 14, 18], although no reproducing kernels were defined or even mentioned therein. We also point out that the authors in [46] had also intended to use this definition for RKBSs, but eventually gave it up due to the example $C([0,1])$. A semi-inner-product structure was imposed in [46] to ensure the existence of a reproducing kernel.

By Definition 2.1, $C([0,1])$ is an RKBS. We shall see what its reproducing kernels look like in Subsection 3.5. The only requirement on RKBSs in our definition is continuity of point evaluations. Definitions of RKBSs in existing literature [38, 43, 46, 47] all impose other requirements to ensure the existence of a reproducing kernel that is not a generalized function. Those requirements more or less seem unnatural. We are able to remove them by exploiting the definition of reproducing kernels via continuous bilinear forms.

A bilinear form between two normed vector spaces $V_{1},V_{2}$ is a function $\langle\cdot,\cdot\rangle_{V_{1}\times V_{2}}$ from $V_{1}\times V_{2}$ to $\mathbb{C}$ that is linear about both arguments. It is said to be continuous if there exists a positive constant $C$ such that

[TABLE]

Definition 2.2

(Reproducing Kernels for RKBS) Let ${\cal B}_{1}$ be an RKBS on a set $\Omega_{1}$. If there exists a Banach space ${\cal B}_{2}$ of functions on another set $\Omega_{2}$, a continuous bilinear form $\langle\cdot,\cdot\rangle_{{\cal B}_{1}\times{\cal B}_{2}}$, and a function $K$ on $\Omega_{1}\times\Omega_{2}$ such that $K(x,\cdot)\in{\cal B}_{2}$ for all $x\in\Omega_{1}$ and

[TABLE]

then we call $K$ a reproducing kernel for ${\cal B}_{1}$. If, in addition, ${\cal B}_{2}$ is also an RKBS on $\Omega_{2}$ and it holds $K(\cdot,y)\in{\cal B}_{1}$ for all $y\in\Omega_{2}$ and

[TABLE]

then we call ${\cal B}_{2}$ an adjoint RKBS of ${\cal B}_{1}$ and call ${\cal B}_{1}$ and ${\cal B}_{2}$ a pair of RKBSs. In this case, $\widetilde{K}(x,y):=K(y,x)$, $x\in\Omega_{2},y\in\Omega_{1}$, is a reproducing kernel for ${\cal B}_{2}$.

We call (2.1) and (2.2) the reproducing properties for the kernel $K$ in RKBSs ${\cal B}_{1}$ and ${\cal B}_{2}$, respectively.

Although there are many conditions in the definition, we shall see that RKBSs and reproducing kernels satisfying the conditions can be easily constructed via a pair of feature maps.

2.2 Constructions via a pair of feature maps

We shall propose a unified framework of constructing RKBSs via a pair of feature maps. We shall discuss in the next section that all existing constructions of RKBSs fall into the framework.

Let $\langle\cdot,\cdot\rangle_{{\cal W}_{1}\times{\cal W}_{2}}$ be a continuous bilinear form $\langle\cdot,\cdot\rangle_{{\cal W}_{1}\times{\cal W}_{2}}$ on Banach spaces ${\cal W}_{1}$ and ${\cal W}_{2}$. We call the linear span $\,{\rm span}\,A$ of a set $A\subseteq{\cal W}_{1}$ dense in ${\cal W}_{1}$ with respect to the bilinear form $\langle\cdot,\cdot\rangle_{{\cal W}_{1}\times{\cal W}_{2}}$ if for any $v\in{\cal W}_{2}$,

[TABLE]

implies $v=0$. Similarly, we can define denseness in ${\cal W}_{2}$ with respect to the bilinear form.

Our construction is described below.

Constructions of RKBSs

Let ${\cal W}_{1},{\cal W}_{2}$ be two Banach spaces, and $\langle\cdot,\cdot\rangle_{{\cal W}_{1}\times{\cal W}_{2}}$ be a continuous bilinear form on ${\cal W}_{1}\times{\cal W}_{2}$. Suppose there exist two nonempty sets $\Omega_{1}$ and $\Omega_{2}$, and mappings

$\Phi_{1}:\Omega_{1}\to{\cal W}_{1},\ \ \Phi_{2}:\Omega_{2}\to{\cal W}_{2}$

such that with respect to the bilinear form

$\,{\rm span}\,\Phi_{1}(\Omega_{1})\mbox{ is dense in }{\cal W}_{1},\ \,{\rm span}\,\Phi_{2}(\Omega_{2})\mbox{ is dense in }{\cal W}_{2}.$

(2.3)

We construct

{\cal B}_{1}:=\Big{\{}f_{v}(x):=\langle\Phi_{1}(x),v\rangle_{{\cal W}_{1}\times{\cal W}_{2}}:v\in{\cal W}_{2},x\in\Omega_{1}\Big{\}}

(2.4)

with norm

$\|f_{v}\|_{{\cal B}_{1}}:=\|v\|_{{\cal W}_{2}}$

and

{\cal B}_{2}:=\Big{\{}g_{u}(y):=\langle u,\Phi_{2}(y)\rangle_{{\cal W}_{1}\times{\cal W}_{2}}:u\in{\cal W}_{1},y\in\Omega_{2}\Big{\}}

(2.5)

with norm

$\|g_{u}\|_{{\cal B}_{2}}:=\|u\|_{{\cal W}_{1}}.$

The construction can be simplified when ${\cal W}_{2}$ is a closed subspace of ${\cal W}_{1}^{*}$, the dual space of continuous linear functionals on ${\cal W}_{1}$. In this case, we always use the natural continuous bilinear form

[TABLE]

The denseness condition (2.3) is satisfied if

[TABLE]

or

[TABLE]

We show that such constructed spaces ${\cal B}_{1}$ and ${\cal B}_{2}$ indeed form a pair of RKBSs, and present the associated reproducing kernel.

Theorem 2.3

Let ${\cal B}_{1}$ and ${\cal B}_{2}$ be constructed as in (2.4) and (2.5), respectively. Then with the bilinear form on ${\cal B}_{1}\times{\cal B}_{2}$

[TABLE]

${\cal B}_{1}$* is an RKBS on $\Omega_{1}$ with the adjoint RKBS ${\cal B}_{2}$ on $\Omega_{2}$. Moreover,*

[TABLE]

is a reproducing kernel for ${\cal B}_{1}$.

Proof: First note that the denseness condition (2.3) guarantees that the $v,u$ in $f_{v}$ and $g_{u}$ are both unique. The definition is hence well-defined. We next prove that ${\cal B}_{1}$ and ${\cal B}_{2}$ are RKBSs. By assumption, there exists a positive constant $C$ such that

[TABLE]

By (2.4) and (2.10), we have for all $f_{v}\in{\cal B}_{1}$ and all $x\in\Omega_{1}$

[TABLE]

Similarly, by (2.5) and (2.10), we have for all $g_{u}\in{\cal B}_{2}$ and all $y\in\Omega_{2}$

[TABLE]

Thus, point evaluation functionals are continuous on both ${\cal B}_{1}$ and ${\cal B}_{2}$. By Definition 2.1, ${\cal B}_{1}$ and ${\cal B}_{2}$ are RKBSs on $\Omega_{1}$ and $\Omega_{2}$, respectively.

Next, $\langle\cdot,\cdot\rangle_{{\cal B}_{1}\times{\cal B}_{2}}$ is a continuous bilinear form as

[TABLE]

Finally, by (2.3), (2.4), (2.8) and (2.9), we have

[TABLE]

for all $x\in\Omega_{1}$ and all $f_{v}\in{\cal B}_{1}$. Similarly,

[TABLE]

for all $y\in\Omega_{2}$ and all $g_{u}\in{\cal B}_{2}$. The proof is complete.   $\Box$

We call $\Phi_{1}:\Omega_{1}\to{\cal W}_{1}$ and $\Phi_{2}:\Omega_{2}\to{\cal W}_{2}$ in Theorem 2.3 a pair of feature maps for the reproducing kernel $K$, and ${\cal W}_{1}$ and ${\cal W}_{2}$ a pair of feature spaces for $K$. The feature spaces and feature maps for an RKBS might not be unique. Different choices of them will lead to different reproducing kernels for the RKBS.

Below we prove the existence of a reproducing kernel for a general RKBS under the mild condition that it is separable.

Theorem 2.4

Every separable RKBS admits a reproducing kernel.

Proof: Let ${\cal B}$ be a separable RKBS on $X$. Thus, there exist $f_{n}\in{\cal B}$, $n\in\mathbb{N}$ such that $\,{\rm span}\,\{f_{n}:n\in\mathbb{N}\}$ is dense in ${\cal B}$. We construct a pair of feature maps and feature spaces for ${\cal B}$ as follows. Let

[TABLE]

Also, set

[TABLE]

Clearly, the denseness condition (2.3) is satisfied. We observe that the space ${\cal B}_{1}$ followed from the construction is exactly ${\cal B}$. By Theorem 2.3,

[TABLE]

is a reproducing kernel for ${\cal B}$.   $\Box$

We remark that the reproducing kernel given in the above theorem is of little use in practice. The purpose of the theorem is to verify that Definition 2.1 is indeed generic for RKBSs. Under the definition, reproducing kernels for RKBSs exist under a very mild condition, namely, separability.

2.3 Orlicz RKBSs

To show the advantages of our framework, we construct a new class of RKBSs based on Orlicz spaces. Orlicz spaces constitute an important generalization of $L^{p}$-spaces. To this end, we shall briefly review the basic theory of Orlicz spaces [29, 30].

Set $\mathbb{R}_{+}:=[0,+\infty)$. Let $\varphi:\mathbb{R}_{+}\to\mathbb{R}_{+}$ be a strictly increasing and continuous function with $\varphi(0)=0$ and $\lim_{t\to+\infty}\varphi(t)=+\infty$. Besides $\varphi$, the following three functions are central to our discussions:

[TABLE]

We call $\Phi$ and $\Psi$ a pair of conjugated nice Young functions. Then

[TABLE]

where equality holds if $x=\psi(y)$, or equivalently $y=\varphi(x)$. The nice Young function $\Psi$ complementary to $\Phi$ can equally be defined by $\Psi(t)=\sup\{x|t|-\Phi(x):x\in\mathbb{R}_{+}\}$, $t\in\mathbb{R}$.

Let $(\Omega,{\cal F},\mu)$ be a measure space. The Orlicz space [30] ${\cal L}^{\Phi}$ consists of all the ${\cal F}$-measurable functions $f:\Omega\to\mathbb{C}$ such that

[TABLE]

It is a linear vector space with two equivalent norms $\|\cdot\|_{\Phi}$, the Gauge norm (Luxemburg norm), and $|\cdot|_{\Phi}$, the Orlicz norm. They are respectively defined by

[TABLE]

and

[TABLE]

They indeed are equivalent as $\Phi(1)\|f\|_{\Phi}\leq|f|_{\Phi}\leq 2\|f\|_{\Phi}$, $f\in{\cal L}^{\Phi}$. If $\varphi(t)=t^{p-1}$ and $\psi(t)=t^{q-1}$, $t\in\mathbb{R}_{+}$, where $1<p,q<+\infty$ with $1/p+1/q=1$, then

[TABLE]

In this case, ${\cal L}^{\Phi}$ is the usual space $L^{p}_{\mu}(\Omega)$ and both the Gauge norm and Orlicz norm equal

[TABLE]

If $\Phi$ and $\Psi$ are a pair of conjugated nice Young functions then $({\cal L}^{\Psi},|\cdot|_{\Psi})=({\cal L}^{\Phi},\|\cdot\|_{\Phi})^{*}$ (see, Theorem 13 in Section 1.2, [30]) in the sense that for each continuous linear functional $T$ on $({\cal L}^{\Phi},\|\cdot\|_{\Phi})$ there exists a unique $g\in{\cal L}^{\Psi}$ such that

[TABLE]

There is a unique semi-inner product ([23], pp. 101–104) on $({\cal L}^{\Phi},\|\cdot\|_{\Phi})$ given by

[TABLE]

where $\,{\rm sgn}\,(t):=t/|t|$ if $t\in\mathbb{C}\setminus\{0\}$ and $\,{\rm sgn}\,(0):=0$. It implies that the standard duality mapping $J_{\Phi}:({\cal L}^{\Phi},\|\cdot\|_{\Phi})\to({\cal L}^{\Psi},|\cdot|_{\Psi})$ has the form

[TABLE]

Particularly, when ${\cal L}^{\Phi}=L^{p}_{\mu}(\Omega)$, we observe that

[TABLE]

which is the usual duality mapping from $L^{p}_{\mu}(\Omega)$ to $L^{q}_{\mu}(\Omega)$ first discovered by Giles [17].

Another interesting example ([30], page 9) is when $\varphi(t)=\log(1+t)$ and $\psi(t)=e^{t}-1$, $t\in\mathbb{R}_{+}$. In this case,

[TABLE]

To emphasize the importance of Orlicz spaces in applications, we give an illustrating example that shows the Orlicz space with the nice Young functions (2.11) can approximate the $\ell^{2}$ or $\ell^{1}$ space. Consider the following sequence of nice Young functions

[TABLE]

Let $\Omega:=\{1,2,\dots,n\}$ and $\mu$ be the counting measure on $\Omega$. Then for each $y:=(y_{1},y_{2},\dots,y_{n})\in\mathbb{C}^{n}$, $\|y\|_{\Phi_{k}}$ equals the smallest nonnegative number $\alpha$ such that

[TABLE]

Hence, the semi-inner-product for the corresponding Orlicz space ${\cal L}^{\Phi_{k}}$ has the form

[TABLE]

Thus, on the unit sphere where $\alpha=1$, we see that

[TABLE]

which is the inner product on the $\ell^{2}$ space, and by the L’Hôpital’s rule that

[TABLE]

which is the semi-inner product on the $\ell^{1}$ space.

We describe our construction of Orlicz RKBSs as follow.

Theorem 2.5

Let $\Phi,\Psi$ be a pair of nice Young functions. Choose

[TABLE]

as a pair of feature spaces. Suppose that there exist feature maps $\Phi_{1}:\Omega_{1}\to{\cal W}_{1},\Phi_{2}:\Omega_{2}\to{\cal W}_{2}$ satisfying the denseness condition (2.3). Then ${\cal B}_{1}$ defined by (2.4) is an RKBS on $\Omega_{1}$ with the adjoint RKBS ${\cal B}_{2}$ defined by (2.5) on $\Omega_{2}$ endowed with the bilinear form (2.8). Moreover, a reproducing kernel for ${\cal B}_{1}$ is

[TABLE]

3 Existing examples of RKBSs

In this section, we aim at justifying that our construction in the previous section covers existing RKBSs in the literature, including the reflexive RKBS [46], the semi-inner-product RKBS [46, 47], the RKBS constructed by Borel measures [37], the RKBS with the $\ell^{1}$ norm [38], the RKBS with positive definite functions [13], and the $p$-norm RKBS [43]. In particular, $C([0,1])$ is included in our framework. Hence, we are able to write out three reproducing kernels for $C([0,1])$.

We shall primarily use Theorem 2.3 to fulfill this task. Let us keep in mind that a pair of RKBSs contain the following eleven ingredients:

[TABLE]

3.1 Reproducing kernel Hilbert spaces

Our construction must include RKHSs. To show this, we shall briefly review basic theory of RKHSs [2, 3, 8, 19, 33, 36, 41, 50].

An RKHS ${\cal H}$ on a nonempty set $X$ is a Hilbert space of certain functions on $X$ such that point evaluation functionals are continuous on ${\cal H}$. By the Riesz representation theorem, there exists a unique reproducing kernel $K$ on $X\times X$ such that

[TABLE]

and

[TABLE]

where $(\cdot,\cdot)_{{\cal H}}$ denotes the inner product on ${\cal H}$. Equation (3.12) is called the reproducing property in machine learning. A function $K$ on $X\times X$ is a reproducing kernel of an RKHS if and only if there exists a feature map $\Phi$ from $X$ to a Hilbert space ${\cal W}$ such that

[TABLE]

Another well-known characterization of a reproducing kernel is for it to be positive semi-definite. There is a bijective correspondence between RKHSs and reproducing kernels. For this sake, the RKHS corresponding to a reproducing kernel $K$ is denoted by ${\cal H}_{K}$. The feature map $\Phi$ and feature space ${\cal W}$ of a reproducing kernel may not be unique. In particular, we say $\Phi$ is the canonical feature map of $K$ if $\Phi(x)=K(\cdot,x)$. In this case, ${\cal W}={\cal H}_{K}$ and

[TABLE]

A reproducing kernel can be easily identified through its feature map. The following result is well-known in machine learning community [44, 50].

Lemma 3.1

[44]** If $K:X\times X\to\mathbb{C}$ is a reproducing kernel with a feature map $\Phi$ from $X$ to a Hilbert space ${\cal W}$, then ${\cal H}_{K}=\{(\Phi(\cdot),u)_{{\cal W}}:u\in{\cal W}\}$ with the inner product

[TABLE]

where $\mathbb{P}_{\Phi}$ denotes the orthogonal projection from ${\cal W}$ onto $\overline{\,{\rm span}\,}\Phi(X)$. If ${\cal W}=\overline{\,{\rm span}\,}\Phi(X)$, then ${\cal H}_{K}$ is isometrically isomorphic to ${\cal W}$ through the linear mapping $(\Phi(\cdot),u)_{{\cal W}}\to u$.

Note that the dual space of ${\cal H}_{K}$ is ${\cal H}_{K}^{*}=\overline{{\cal H}_{K}}:=\{\bar{f}:f\in{\cal H}_{K}\}$. By Theorem 2.3 and Lemma 3.1, we obtain the following result.

Example 3.2

Let ${\cal H}_{K}$ be an RKHS on $X$ with the reproducing kernel $K$, and let $\Phi$ from $X$ to a Hilbert space ${\cal W}$ be a feature map of $K$ such that ${\cal W}=\overline{\,{\rm span}\,}\Phi(X)$. Choose

[TABLE]

[TABLE]

[TABLE]

in Theorem 2.3. Then

[TABLE]

with norm $\|f_{v}\|_{{\cal B}_{1}}:=\|\bar{v}\|_{{\cal W}}$ is an RKBS on $\Omega_{1}$. Its adjoint RKBS is

[TABLE]

with norm $\|g_{u}\|_{{\cal B}_{2}}:=\|u\|_{{\cal W}}$. The bilinear form on ${\cal B}_{1}\times{\cal B}_{2}$ is given by

[TABLE]

Moreover,

[TABLE]

is a reproducing kernel for ${\cal B}_{1}$.

Proof: The denseness condition (2.6) is satisfied. By Theorem 2.3, the proof is complete.   $\Box$

Therefore, we have showed that an RKHS is an RKBS with the same reproducing kernel.

3.2 Reflexive RKBSs

Learning in Banach spaces has received considerable attention in the past two decades. The notion of reproducing kernel Banach spaces was introduced in machine learning in 2009, [46]. Reflexive RKBSs constructed in [46] are the first class of RKBSs with reproducing kernels.

A normed vector space $V$ is reflexive if $(V^{*})^{*}=V$, that is, every continuous linear functional $T$ on $V^{*}$ must be of the form

[TABLE]

for some $u\in V$. A Banach space is reflexive if and only if its dual is reflexive, [24].

The definition and construction of reflexive RKBSs in [46] are described below.

Definition of Reflexive RKBSs in [46]. An RKBS on $X$ is a reflexive Banach space ${\cal B}$ of functions on $X$ for which ${\cal B}^{*}$ is isometrically isomorphic to a Banach space ${\cal B}^{\#}$ of functions on $X$ and the point evaluation is continuous on both ${\cal B}$ and ${\cal B}^{\#}$.

Construction of Reflexive RKBSs in [46]. Let ${\cal W}$ be a reflexive Banach space. Suppose that there exists $\Phi:X\to{\cal W}$, and $\Phi^{*}:X\to{\cal W}^{*}$ such that $\overline{\,{\rm span}\,}\Phi(X)={\cal W}$, $\overline{\,{\rm span}\,}\Phi^{*}(X)={\cal W}^{*}$. Then

[TABLE]

is an RKBS on $X$ with the dual space

[TABLE]

and the bilinear form

[TABLE]

Moreover, $K(x,y):=\langle\Phi(x),\Phi^{*}(y)\rangle_{{\cal W}}$ is a reproducing kernel for ${\cal B}$.

We show that the above construction of reflexive RKBSs falls within our framework.

Example 3.3

Assume the same conditions as in the above construction of reflexive RKBSs. Choose

[TABLE]

[TABLE]

[TABLE]

in Theorem 2.3. Then

[TABLE]

with norm $\|f_{v}\|_{{\cal B}_{1}}:=\|v\|_{{\cal W}}$ is an RKBS on $\Omega_{1}$. Its adjoint RKBS is

[TABLE]

with norm $\|g_{u}\|_{{\cal B}_{2}}:=\|u\|_{{\cal W}^{*}}$. The bilinear form on ${\cal B}_{1}\times{\cal B}_{2}$ takes the form

[TABLE]

Moreover, a reproducing kernel of ${\cal B}_{1}$ is

[TABLE]

Proof: Note that ${\cal W}$ is a reflexive Banach space. The denseness condition (2.6) is hence satisfied. By Theorem 2.3, the proof is complete.   $\Box$

3.3 Semi-inner-product RKBSs

As mentioned in [46], the lack of an inner product may cause arbitrariness in the properties of the associated reproducing kernel of an RKBS. To overcome this, the authors also proposed the second class of RKBSs, namely, semi-inner-product RKBSs. Thanks to the tool of semi-inner-products, existence, uniqueness and representer theorems for the standard learning schemes in semi-inner-product RKBSs were established therein.

For a uniformly convex and uniformly Fréchet differentiable Banach space ${\cal B}$, there exists a unique semi-inner product $[\cdot,\cdot]_{{\cal B}}:{\cal B}\times{\cal B}\to\mathbb{C}$ such that for all $f,g,h\in{\cal B}$ and $\alpha\in\mathbb{C}$,

(i)

$[f+g,h]_{{\cal B}}=[f,h]_{{\cal B}}+[g,h]_{{\cal B}}$,  $[\alpha f,g]_{{\cal B}}=\alpha[f,g]_{{\cal B}}$,

(ii)

$[f,f]_{{\cal B}}>0$ for $f\neq 0$,

(iii)

(The Cauchy-Schwartz inequality) $|[f,g]_{{\cal B}}|^{2}\leq[f,f]_{{\cal B}}[g,g]_{{\cal B}}$.

We refer to Section 2 in [48] or Section 5.5 in [24], and [10, 17, 22, 46] for more details on semi-inner products. By the Cauchy-Schwartz inequality, for each $g\in{\cal B}$, $f\to[f,g]_{{\cal B}}$ is a bounded linear functional on ${\cal B}$, which is denoted by $g^{*}\in{\cal B}^{*}$ and called the dual element of $g$. Following this definition, we have

[TABLE]

Giles [17] proved that if ${\cal B}$ is a uniformly convex and uniformly Fréchet differentiable Banach space ${\cal B}$, then the duality mapping $f\to f^{*}$ is bijective from ${\cal B}$ to ${\cal B}^{*}$. Moreover,

[TABLE]

is a semi-inner product on ${\cal B}^{*}$.

The following definition of the semi-inner-product RKBS (s.i.p. RKBS) comes from [46].

Definition of s.i.p. RKBS in [46]. We call a uniformly convex and uniformly Fréchet differentiable Banach space of functions on X an s.i.p. RKBS.

Note that a uniformly convex Banach space is reflexive. It follows that an s.i.p. RKBS is also a reflexive RKBS. We are able to construct the s.i.p. RKBS in our framework.

Example 3.4

Let ${\cal W}$ be a uniformly convex and uniformly Fréchet differentiable Banach space, and $\Phi:X\to{\cal W}$ and $\Phi^{*}:X\to{\cal W}^{*}$ be such that $\overline{\,{\rm span}\,}\Phi(X)={\cal W}$ and $\overline{\,{\rm span}\,}\Phi^{*}(X)={\cal W}^{*}$. Choose

[TABLE]

[TABLE]

[TABLE]

in Theorem 2.3. Then

[TABLE]

with norm $\|f_{v}\|_{{\cal B}_{1}}:=\|v\|_{{\cal W}}$ is an RKBS on $\Omega_{1}$. Its adjoint RKBS is

[TABLE]

with norm $\|g_{u^{*}}\|_{{\cal B}_{2}}:=\|u^{*}\|_{{\cal W}^{*}}$. The bilinear form on ${\cal B}_{1}\times{\cal B}_{2}$ is given by

[TABLE]

Moreover, a reproducing kernel for ${\cal B}_{1}$ is

[TABLE]

Proof: By assumptions, ${\cal W}$ is a reflexive Banach space. The denseness condition (2.6) is hence satisfied. By Theorem 2.3, the proof is complete.   $\Box$

Here, we shall point out in Example 3.4 that

[TABLE]

We remark that ${\cal B}_{1}$ and ${\cal B}_{2}$ in Example 3.4 are exactly the ${\cal B}$ and ${\cal B}^{*}$ constructed in (Theorem 10, [46]). They also have the same reproducing kernel.

3.4 RKBSs by Borel measures

In order to improve the learning rate estimate of the $\ell_{1}$-regularized least square regression, a class of RKBSs by Borel measures was constructed in [37] to have the $\ell_{1}$ norm and satisfy the linear representer theorem.

Suppose that $X$ is a locally compact Hausdorff space and denote by $C_{0}(X)$ the Banach space of continuous functions $f:X\to\mathbb{C}$ such that for all $\varepsilon>0$, the set $\{x\in X:|f(x)|\geq\varepsilon\}$ is compact. Its dual space is isometrically isomorphic to the space ${\cal M}(X)$ of all the regular complex-valued Borel measures on $X$ (see, Theorem 6.19 in [31]). The norm of each measure $v\in{\cal M}(X)$ is its total variation $\|v\|_{TV}$.

We are able to construct the RKBSs by Borel measures in our framework.

Example 3.5

Let $X$ be a locally compact Hausdorff space, and let $K:X\times X\to\mathbb{C}$ be a continuous function such that $\overline{\,{\rm span}\,}\{K(\cdot,x):x\in X\}=C_{0}(X)$. Choose

[TABLE]

[TABLE]

[TABLE]

in Theorem 2.3. Then

[TABLE]

with norm $\|f_{v}\|_{{\cal B}_{1}}:=\|v\|_{TV}$ is an RKBS on $\Omega_{1}$. Its adjoint RKBS is

[TABLE]

with norm $\|g_{u}\|_{{\cal B}_{2}}:=\sup_{y\in\Omega_{2}}|u(y)|$. The bilinear form on ${\cal B}_{1}\times{\cal B}_{2}$ is given by

[TABLE]

Moreover, $K$ is a reproducing kernel for ${\cal B}_{1}$.

Proof: As $\Phi_{2}(X)=\{\delta_{y}:y\in X\}$ is dense in ${\cal M}(X)=(C_{0}(X))^{*}$ under the weak* topology, the denseness condition (2.7) holds true. By Theorem 2.3, the proof is complete.   $\Box$

3.5 RKBSs with the $\ell^{1}$ norm

The RKBS with the $\ell^{1}$ norm was developed in [38] for sparse learning. The construction starts directly with a kernel function satisfying the following two requirements:

(i)

$K$ is bounded,

(ii)

for all pairwise distinct sampling points $x_{j}\in X$, $j\in\mathbb{N}$ and $c=(c_{j}:j\in\mathbb{N})\in\ell^{1}(\mathbb{N})$, $\sum_{j\in\mathbb{N}}c_{j}K(x_{j},x)=0$ for all $x\in X$ implies $c=0$.

Denote for any nonempty set $X$ by

[TABLE]

the Banach space of functions on $X$ that is integrable with respect to the counting measure on $X$. Note that $X$ might be uncountable but for any $u\in\ell^{1}(X)$, $\,{\rm supp}\,u:=\{x\in X:u_{x}\neq 0\}$ must be at most countable. Note that $\ell^{1}(X)$ can be imbedded into ${\cal M}(X)$.

Construction of RKBSs with the $\ell^{1}$ Norm in [38]. Let $K:X\times X\to\mathbb{C}$ be a kernel satisfying aforementioned two requirements (i) and (ii). Then

[TABLE]

with norm $\|f_{u}\|_{{\cal B}}:=\|u\|_{\ell^{1}}$, and ${\cal B}^{\#}$, the completion of the vector space of functions $\sum_{j=1}^{n}v_{j}K(\cdot,y_{j})$, $y_{j}\in X$ under the supremum norm

[TABLE]

are both Banach space of functions on $X$ where point evaluations are continuous linear functionals. In addition, the bilinear form

[TABLE]

can be extended to ${\cal B}\times{\cal B}^{\#}$ such that

[TABLE]

and

[TABLE]

We show below that RKBSs with the $\ell^{1}$ norm fall into our framework.

Example 3.6

Let $X$ be a locally compact Hausdorff space, and let $K:X\times X\to\mathbb{C}$ be bounded and continuous such that $C_{0}(X)=\overline{\,{\rm span}\,}\{K(\cdot,x):x\in X\}$. Choose

[TABLE]

[TABLE]

[TABLE]

in Theorem 2.3. Then

[TABLE]

with norm $\|f_{v}\|_{{\cal B}_{1}}:=\|v\|_{\ell^{1}(X)}$ is an RKBS on $\Omega_{1}$. Its adjoint RKBS is

[TABLE]

with norm $\|g_{u}\|_{{\cal B}_{2}}:=\sup_{y\in X}|u(y)|$. The bilinear form on ${\cal B}_{1}\times{\cal B}_{2}$ is given by

[TABLE]

Moreover, $K$ is a reproducing kernel for ${\cal B}_{1}$.

Proof: The boundedness of $K$ guarantees that $f_{v}$ in ${\cal B}_{1}$ is well-defined. The set $\Phi_{2}(X)=\{\delta_{y}:y\in X\}$ is dense in ${\cal M}(X)=(C_{0}(X))^{*}$ under the weak* topology. The denseness condition (2.7) is hence satisfied. By Theorem 2.3, the proof is complete.   $\Box$

In the rest of this subsection, we discuss the particular and interesting space $C([0,1])$. We shall show that $C([0,1])$ is an RKBS in our framework and shall present several explicit reproducing kernels for the space. Non-uniqueness of the reproducing kernel is caused by existence of many feature maps $\Phi_{1}:[0,1]\to C([0,1])$ satisfying the denseness condition $\overline{\,{\rm span}\,}{\Phi_{1}(X)}=C([0,1])$.

Example 3.7

Choose

[TABLE]

[TABLE]

[TABLE]

in Theorem 2.3. Then

[TABLE]

with norm $\|f_{v}\|_{{\cal B}_{1}}:=\|v\|_{TV}$ is an RKBS on $\Omega_{1}$. Its adjoint RKBS is

[TABLE]

with norm $\|g_{u}\|_{{\cal B}_{2}}:=\sup_{y\in[0,1]}|u(y)|$. The bilinear form on ${\cal B}_{1}\times{\cal B}_{2}$ is given by

[TABLE]

Moreover, $K(x,y):=\langle\Phi_{1}(x),\Phi_{2}(y)\rangle_{{\cal W}_{1}}=1-|x-y|$, $x,y\in[0,1]$ is a reproducing kernel for ${\cal B}_{1}$.

Proof: Note that every continuous function on $[0,1]$ can be approximated uniformly by piecewise linear functions. Thus, we have ${\cal W}_{1}=\overline{\,{\rm span}\,}\Phi_{1}([0,1])=\overline{\,{\rm span}\,}\{1-|\cdot-x|:x\in[0,1]\}$. The denseness condition (2.7) is hence satisfied. By Theorem 2.3, the proof is complete.   $\Box$

We can also take $\Phi_{1}(x)(t):=(1+t)^{x}$ or $\Phi_{1}(x)(t):=e^{tx}$ in Example 3.7 and they all satisfy the denseness condition (2.6). This is true by the fact in complex analysis that zeros of a nontrivial holomorphic function are isolated. Correspondingly,

[TABLE]

can be viewed as reproducing kernels for ${\cal B}_{1}$ as well. By Definition 2.2, $\widetilde{K}(x,y)=K(y,x)$ is a reproducing kernel of ${\cal B}_{2}=C([0,1])$. Thus, we obtain three reproducing kernels for $C([0,1])$:

[TABLE]

3.6 RKBSs with positive definite functions

The relationship between generalized Sobolev spaces and RKHSs was established by developing a connection between Green functions and reproducing kernels. Motivated by this, the authors in [13] used Fourier transform techniques to construct RKBSs with positive definite functions. Furthermore, Ye in [45] developed numerical algorithms for support vector machines in those RKBSs.

Let $\phi\in L^{1}(\mathbb{R}^{d})\cap C(\mathbb{R}^{d})$ be positive definite, that is, for all finite distinct points $x_{1},x_{2},\ldots,x_{n}\in\mathbb{R}^{d}$, the matrix $[\phi(x_{j}-x_{k}):j,k=1,2,\dots,n]$ is strictly positive-definite. It has been known that $\phi$ is positive definite if and only if it is bounded, its Fourier transform $\hat{\phi}$ is nonnegative, and

[TABLE]

has positive Lebesgue measure (see, for instance, Section 6.2 in [41]). In this paper, we use the following form of the Fourier transform

[TABLE]

and the inverse Fourier transform

[TABLE]

We show below that RKBSs (Theorem 4.1 in [13] or Theorem 1 in [45]) with positive definite functions fall into our framework.

Example 3.8

Let $1<q\leq 2\leq p<+\infty$ and $1/p+1/q=1$. Suppose that $\phi\in L^{1}(\mathbb{R}^{d})\cap C(\mathbb{R}^{d})$ is a positive definite, even function on $\mathbb{R}^{d}$ and that $\hat{\phi}^{q-1}\in L^{1}(\mathbb{R}^{d})$. Choose

[TABLE]

[TABLE]

[TABLE]

[TABLE]

[TABLE]

and define the following continuous bilinear form on ${\cal W}_{1}\times{\cal W}_{2}$

[TABLE]

in Theorem 2.3. Then

[TABLE]

with norm $\|f_{v}\|_{{\cal B}_{1}}:=\|v\|_{{\cal W}_{2}}$ is an RKBS on $\Omega_{1}$. Its adjoint RKBS is

[TABLE]

with norm $\|g_{u}\|_{{\cal B}_{2}}:=\|u\|_{{\cal W}_{1}}$. The bilinear form on ${\cal B}_{1}\times{\cal B}_{2}$ is defined by

[TABLE]

Moreover, $\phi(x-y)$ is a reproducing kernel for ${\cal B}_{1}$.

Proof: To begin with, we shall check the denseness condition (2.3). Since $\phi\in L^{1}(\mathbb{R}^{d})\cap C(\mathbb{R}^{d})$ is positive definite, we have $\hat{\phi}\in L^{1}(\mathbb{R}^{d})\cap C(\mathbb{R}^{d})$ and $\hat{\phi}^{p-1}\in L^{1}(\mathbb{R}^{d})$ for $p\geq 2$. Observe that $\check{\phi}=\hat{\phi}$ as $\phi$ is an even function on $\mathbb{R}^{d}$. For each $x\in\mathbb{R}^{d}$, we compute

[TABLE]

which implies $\phi(\cdot-x)\in{\cal W}_{1}$ for all $x\in\mathbb{R}^{d}$. Furthermore, for any $v\in{\cal W}_{2}$,

[TABLE]

implies $v=0$. Thus, $\,{\rm span}\,\Phi_{1}(\Omega_{1})=\,{\rm span}\,\{\phi(\cdot-x):x\in\mathbb{R}^{d}\}$ is dense in ${\cal W}_{1}$ with respect to the bilinear form $\langle\cdot,\cdot\rangle_{{\cal W}_{1}\times{\cal W}_{2}}$. That $\,{\rm span}\,\Phi_{2}(\Omega_{2})$ is dense in ${\cal W}_{2}$ with respect to the bilinear form can be proved in a similar manner. The denseness condition (2.3) is hence satisfied.

Let $K(x,y):=\langle\Phi_{1}(x),\Phi_{2}(y)\rangle_{{\cal W}_{1}\times{\cal W}_{2}}=\langle\phi(\cdot-x),\phi(\cdot-y)\rangle_{{\cal W}_{1}\times{\cal W}_{2}}$. By (3.13), we have

[TABLE]

The proof is complete.   $\Box$

We remark that the space

[TABLE]

in Example 3.8 is ${\cal B}_{\Phi}^{p}(\mathbb{R}^{d})$, $2\leq p<+\infty$ in Theorem 1 of [45].

3.7 $p$-norm RKBSs

The use of semi-inner products in the construction of RKBSs has its limitations. To overcome this issue and for the sake of sparse learning, Xu and Ye [43] constructed a class of $p$-norm RKBSs via generalized Mercer kernels.

The generalized Mercer kernel used in [43] takes the following form

[TABLE]

where $\{\phi_{n}:n\in\mathbb{Z}\}$ and $\{\psi_{n}:n\in\mathbb{Z}\}$ are sequences of functions on $\Omega_{1}$ and $\Omega_{2}$, respectively. For instance, the Gaussian kernel and the Brownian bridge kernel are generalized Mercer kernels. Denote by $c_{0}$ the Banach space of sequences on $\mathbb{Z}$ that vanish at infinity and are endowed with the supremum norm. We show below that $p$-norm RKBSs fall into our framework.

Example 3.9

Let $1<p<+\infty$, $1/p+1/q=1$. Choose

[TABLE]

[TABLE]

[TABLE]

in Theorem 2.3 such that ${\cal W}_{1}=\overline{\,{\rm span}\,}\Phi_{1}(\Omega_{1})$ and ${\cal W}_{2}=\overline{\,{\rm span}\,}\Phi_{2}(\Omega_{2})$. Then

[TABLE]

with norm $\|f_{v}\|_{{\cal B}_{1}}:=\|v\|_{\ell^{p}}$ is an RKBS on $\Omega_{1}$. Its adjoint RKBS is

[TABLE]

with norm $\|g_{u}\|_{{\cal B}_{2}}:=\|u\|_{\ell^{q}}$. The bilinear form on ${\cal B}_{1}\times{\cal B}_{2}$ is defined by

[TABLE]

Moreover, $K$ defined as in (3.14) is a reproducing kernel for ${\cal B}_{1}$.

Proof: Note that $\ell^{p}=(\ell^{q})^{*}$, where $1/{p}+1/{q}=1$, $1<p<+\infty$. The denseness condition (2.6) is hence satisfied. The claim follows from Theorem 2.3.   $\Box$

The RKBSs ${\cal B}_{1}$ and ${\cal B}_{2}$ in Example 3.9 are exactly ${\cal B}_{K}^{p}(\Omega_{1})$ and ${\cal B}^{q}_{K^{\prime}}(\Omega_{2})$ defined as in (3.5) and (3.6) of [43], respectively. Here, $K^{\prime}(x,y):=K(y,x)$, $x\in\Omega_{2},y\in\Omega_{1}$.

Example 3.10

Choose

[TABLE]

[TABLE]

[TABLE]

in Theorem 2.3 such that ${\cal W}_{1}=\overline{\,{\rm span}\,}\Phi_{1}(\Omega_{1})$ and ${\cal W}_{2}=\overline{\,{\rm span}\,}\Phi_{2}(\Omega_{2})$. Then

[TABLE]

with norm $\|f_{v}\|_{{\cal B}_{1}}:=\|v\|_{\ell^{1}}$ is an RKBS on $\Omega_{1}$. Its adjoint RKBS is

[TABLE]

with norm $\|g_{u}\|_{{\cal B}_{2}}:=\|u\|_{c_{0}}$. The bilinear form on ${\cal B}_{1}\times{\cal B}_{2}$ is defined by

[TABLE]

Moreover, $K$ defined as in (3.14) is a reproducing kernel for ${\cal B}_{1}$.

Proof: Note that ${\cal W}_{2}=\ell^{1}={\cal W}_{1}^{*}=c_{0}^{*}$. The denseness condition (2.6) is hence satisfied. By Theorem 2.3, the proof is complete.   $\Box$

The RKBSs ${\cal B}_{1}$ and ${\cal B}_{2}$ in Example 3.10 are exactly ${\cal B}_{K}^{1}(\Omega_{1})$ and ${\cal B}^{\infty}_{K^{\prime}}(\Omega_{2})$ defined as in (3.14) and (3.15) of [43], respectively.

4 Representer theorems for machine learning in RKBSs

Most machine learning tasks boil down to a regularized minimization problem. When kernel methods are used, a representer theorem asserts that the minimizer is a linear combination of the kernel functions at the sampling points. This is key to the mathematical analysis of kernel methods in machine learning [7, 8, 33]. The classical representer theorem in RKHSs was first established by Kimeldorf and Wahba [21]. The result was generalized to non-quadratic loss functions in [6], and to general regularizers in [32]. Recent references [38, 43, 45, 46, 48] developed representer theorems for various RKBSs introduced in the previous section. The primary purpose of this section is to present a representer theorem for RKBSs constructed in our framework, thus unifying the representer theorems in the references.

Let ${\cal B}_{1}$ be an RKBS constructed as in Theorem 2.3 via a continuous bilinear form and a pair of feature maps. We shall establish a representer theorem for the regularization network in ${\cal B}_{1}$. We begin the analysis with the related minimal norm interpolation in ${\cal B}_{1}$.

4.1 Minimal norm interpolation

The problem of minimal norm interpolation is to find a function with the smallest norm in ${\cal B}_{1}$ that interpolates a prescribed set of sampled data. Let $\mathbb{N}_{m}:=\{1,2,\dots,m\}$, ${\bf x}:=\{x_{j}:j\in\mathbb{N}_{m}\}\subseteq\Omega_{1}$ be a set of $m$ pairwise distinct inputs, and ${\bf t}:=\{t_{j}:j\in\mathbb{N}_{m}\}\subseteq\mathbb{C}$ be the corresponding outputs.

The minimal norm interpolation problem looks for the minimizer

[TABLE]

provided that it exists and is unique. By (2.4) and the denseness condition (2.3), (4.15) can be equivalently reformulated as

[TABLE]

where

[TABLE]

with

[TABLE]

In the special case when $t_{j}=0$ for every $1\leq j\leq m$,

[TABLE]

where $\Phi_{1}({\bf x}):=\{\Phi_{1}(x_{1}),\Phi_{1}(x_{2}),\dots,\Phi_{1}(x_{m})\}$. Here, for a subset $A\subseteq{\cal W}_{1}$,

[TABLE]

Recall that for a subset $A$ in a normed vector space $V$,

[TABLE]

When ${\cal W}_{2}\subseteq{\cal W}_{1}^{*}$, $A^{\vdash}\subseteq A^{\perp}$ for $A\subseteq{\cal W}_{1}$.

Next, we shall explore the condition ensuring that $V_{{\bf x},{\bf t}}$ is nonempty.

Lemma 4.1

The set $V_{{\bf x},{\bf t}}$ defined by (4.17) is nonempty for any ${\bf t}\in\mathbb{C}^{m}$ if and only if $\{\Phi_{1}(x_{j}):j\in\mathbb{N}_{m}\}$ is linearly independent in ${\cal W}_{1}$.

Proof: One sees that $V_{{\bf x},{\bf t}}$ is nonempty for any ${\bf t}\in\mathbb{C}^{m}$ if and only if $\,{\rm span}\,\{(f(x_{j}):j\in\mathbb{N}_{m}):f\in{\cal B}_{1}\}$ is dense in $\mathbb{C}^{m}$. Note that for each $f\in{\cal B}_{1}$, there exists a unique $v\in\overline{\,{\rm span}\,}\{\Phi_{2}(y):y\in\Omega_{2}\}$ such that $f(x):=f_{v}(x)=\langle\Phi_{1}(x),v\rangle_{{\cal W}_{1}\times{\cal W}_{2}}$, $x\in\Omega_{1}$. Using the reproducing property, we have for each $(c_{j}:j\in\mathbb{N}_{m})\in\mathbb{C}^{m}$ that

[TABLE]

By the denseness condition (2.3), the above equation implies that $\{\Phi_{1}(x_{j}):j\in\mathbb{N}_{m}\}$ is linearly independent in ${\cal W}_{1}$ if and only if $\,{\rm span}\,\{(f(x_{j}):j\in\mathbb{N}_{m}):f\in{\cal B}_{1}\}$ is dense in $\mathbb{C}^{m}$. The proof is complete.   $\Box$

Before moving on, we need several concepts from the theory of Banach spaces (see, for instance, Sections 1.11, 5.1 and 5.4 in [24]). A normed vector space $V$ is strictly convex (rotund) if $\|tf+(1-t)g\|_{V}<1$ whenever $\|f\|_{V}=\|g\|_{V}=1$, $f\neq g$, and $0<t<1$, and is Gâteaux differentiable if for all $f,h\in V\setminus\{0\}$, $\lim_{\tau\to 0}\frac{\|f+\tau h\|_{V}-\|f\|_{V}}{\tau}$ exists. For each $f\neq 0$ in a Gâteaux differentiable normed vector space $V$, there exists a bounded linear functional, denoted by ${\cal G}(f)\in V^{*}$ and called a Gâteaux derivative of $f$, such that

[TABLE]

We make a convention that ${\cal G}(f)=0$ if $f=0$.

Reflexivity and strict convexity of a Banach space ensure existence and uniqueness of the best approximation in the space (see, Corollary 5.1.19 in [24]).

Lemma 4.2

[24]** If $V$ is a reflexive and strictly convex Banach space, then for any nonempty closed convex subset $A\subseteq V$ and any $x\in V$ there exists a unique $x_{0}\in A$ such that

[TABLE]

The last lemma needed is about orthogonality in normed vector spaces (see, page 272, [20]). Let $V$ be a normed vector space. We say that $f\in V$ is orthogonal to $g\in V$ if $\|f+\tau g\|_{V}\geq\|f\|_{V}$ for all $\tau\in\mathbb{C}$. We call $f\in V$ orthogonal to a subspace ${\cal N}$ of $V$ if it is orthogonal to every vector in ${\cal N}$.

Lemma 4.3

[20]** If a normed vector space $V$ is Gâteaux differentiable, then $f\in V$ is orthogonal to $g\in V$ if and only if $\langle g,{\cal G}(f)\rangle_{V}=0$.

We are now ready to develop a representer theorem for the minimal norm interpolation in RKBSs constructed in our framework.

Theorem 4.4

(Representer theorem) Assume the same assumptions as in Theorem 2.3. In addition, suppose that ${\cal W}_{2}$ is reflexive, strictly convex and Gâteaux differentiable, and the set $\{\Phi_{1}(x_{j}):j\in\mathbb{N}_{m}\}$ is linearly independent in ${\cal W}_{1}$. Then the minimal norm interpolation problem (4.16) has a unique solution $v_{\inf}\in{\cal W}_{2}$ and it satisfies

[TABLE]

Proof: By Lemma 4.1, linear independence of $\{\Phi_{1}(x_{j}):j\in\mathbb{N}_{m}\}$ in ${\cal W}_{1}$ implies that $V_{{\bf x},{\bf t}}$ is nonempty. Clearly, $V_{{\bf x},{\bf t}}$ is closed and convex in ${\cal W}_{2}$. By Lemma 4.2, there exists a unique $v\in V_{{\bf x},{\bf t}}$, denoted by $v_{\inf}$, such that

[TABLE]

If $v_{\inf}=0$ in ${\cal W}_{2}$, then (4.19) holds. Observe that $v_{\inf}+V_{{\bf x},0}=V_{{\bf x},{\bf t}}$, where $V_{{\bf x},0}$ is defined by (4.18). Since $v_{\inf}$ is the minimizer for the minimal norm interpolation problem (4.16) and $v_{\inf}+v\in V_{{\bf x},{\bf t}}$ for each $v\in V_{{\bf x},0}$, we have

[TABLE]

By Lemma 4.3, $\langle v,{\cal G}(v_{\inf})\rangle_{{\cal W}_{2}}=0$ for all $v\in V_{{\bf x},0}$, which implies ${\cal G}(v_{\inf})\in V_{{\bf x},0}^{\perp}$. The proof is complete.   $\Box$

When ${\cal W}_{2}={\cal W}_{1}^{*}$, the above result can be simplified.

Corollary 4.5

Assume the same assumptions as in Theorem 4.4. If, in addition, ${\cal W}_{2}={\cal W}_{1}^{*}$, then the minimal norm interpolation problem (4.16) has a unique solution $v_{\inf}\in{\cal W}_{2}$ and it satisfies

[TABLE]

Proof: Since ${\cal W}_{2}={\cal W}_{1}^{*}$ and ${\cal W}_{2}$ is reflexive, by Theorem 4.4, we have

[TABLE]

The proof is complete.   $\Box$

We shall make some comments on the assumptions of Corollary 4.5. Firstly, note that the constructed spaces ${\cal B}_{1}$ and ${\cal B}_{2}$ are isomorphic to ${\cal W}_{2}$ and ${\cal W}_{1}$, respectively. Therefore, ${\cal W}_{2}={\cal W}_{1}^{*}$ is equivalent to ${\cal B}_{2}={\cal B}_{1}^{*}$ (in the sense of isomorphism). By the discussion in Section 3, the condition ${\cal W}_{2}={\cal W}_{1}^{*}$ is satisfied by RKHSs, Orlicz RKBSs, reflexive RKBSs, semi-inner-product RKBSs, RKBSs with Borel measures, RKBSs with positive definite functions, and the $p$-norm RKBSs ($1<p<+\infty$), but not satisfied by RKBSs with the $\ell^{1}$ norm or the 1-norm RKBSs. Secondly, for $1<p<+\infty$, $\ell^{p}$ and $L^{p}$ are reflexive, strictly convex, and Gâteaux differentiable. It is also well-known that $\ell^{1}$ is non-reflexive. As a result, the properties of reflexivity, strict convexity, and Gâteaux differentiability are satisfied by the semi-inner product RKBSs, the $p$-norm RKBSs ($1<p<+\infty$), and RKBSs with positive definite functions, but are not satisfied by the RKBSs with Borel measures, RKBSs with the $\ell^{1}$ norm, or the 1-norm RKBSs [43]. Consequently, additional requirements have to be imposed to ensure a linear representer theorem in the latter three spaces. For instance, a uniform boundedness of the Lebesgue constant condition on the reproducing kernel was imposed in [38] for the RKBS with the $\ell^{1}$-norm.

In addition, we remark that when an RKBS reduces to an RKHS, the above results recover the classical representer theorem for minimal norm interpolation in RKHSs.

4.2 Regularization networks

We consider learning a function from a prescribed set of finite sampling data

[TABLE]

Let $L_{{\bf t}}:\mathbb{C}^{m}\to\mathbb{R}_{+}$ be a loss function that is continuous and convex. For each $f\in{\cal B}_{1}$, we set

[TABLE]

where $f({{\bf x}}):=(f(x_{j}):j\in\mathbb{N}_{m})$, and the regularization function $\phi:\mathbb{R}_{+}\to\mathbb{R}_{+}$ is continuous, convex and strictly increasing with $\lim_{t\to+\infty}\phi(t)=+\infty$. A regularization network in an RKBS ${\cal B}_{1}$ takes the form:

[TABLE]

Note that each $f\in{\cal B}_{1}$ corresponds to a unique $v\in\overline{\,{\rm span}\,}\{\Phi_{2}(y):y\in\Omega_{2}\}$ such that

[TABLE]

Thus, (4.20) reduces to

[TABLE]

In order to prove the existence of the minimizer for the regularization network, we need the following result (see, Proposition 6, page 75, [11]).

Lemma 4.6

[11]** Let $V$ be a reflexive Banach space and $F:V\to\mathbb{R}\cup\{+\infty\}$ be convex and lower semi-continuous. If there is an $M\in\mathbb{R}$ such that the set $\{v\in V:F(v)\leq M\}$ is nonempty and bounded, then $F$ attains its minimum on $V$.

Next, we establish a representer theorem for the regularization network.

Theorem 4.7

(Representer theorem) Assume the same assumptions as in Theorem 4.4. Then the regularization network (4.20) possesses a unique solution $f_{{\bf v}_{\inf}}$ where ${\bf v}_{\inf}\in{\cal W}_{2}$ satisfies

[TABLE]

Proof: We first prove the uniqueness by contradiction. Assume that there are two different minimizers $f_{1},f_{2}\in{\cal B}_{1}$ for (4.20). Let $f_{3}:=\frac{1}{2}(f_{1}+f_{2})$. Since ${\cal W}_{2}$ is reflexive, strictly convex and Gâteaux differentiable, ${\cal B}_{1}$ is reflexive, strictly convex and Gâteaux differentiable as well. By the strict convexity of ${\cal B}_{1}$, we have

[TABLE]

By assumptions on $L_{{\bf t}}$ and $\phi$, it follows that

[TABLE]

contradicting that $f_{1}$ is a minimizer.

Next, we shall show the existence. If $f\in{\cal B}_{1}$ satisfies $\|f\|_{{\cal B}_{1}}>\phi^{-1}(\frac{{\cal E}_{{\bf z},\lambda}(0)}{\lambda})$ then

[TABLE]

Thus,

[TABLE]

Clearly, $E$ is nonempty and bounded in the reflexive Banach space ${\cal B}_{1}$. Observe that ${\cal E}_{{\bf z},\lambda}$ is convex and continuous on ${\cal B}_{1}$. By Lemma 4.6, ${\cal E}_{{\bf z},\lambda}$ attains its minimum on ${\cal B}_{1}$.

Finally, suppose that $f_{v}=\langle\Phi_{1}(\cdot),v\rangle_{{\cal W}_{1}\times{\cal W}_{2}}\in{\cal B}_{1}$ is the minimizer for (4.20). We set

[TABLE]

By Theorem 4.4, there exists a unique solution $v_{\inf}\in{\cal W}_{2}$ for the minimal norm interpolation (4.15) with the samples $D$, and it satisfies (4.19). It follows that $f_{v_{\inf}}=\langle\Phi_{1}(\cdot),v_{\inf}\rangle_{{\cal W}_{1}\times{\cal W}_{2}}$ interpolates the sample data $D$ and for all $v\in{\cal W}_{2}$

[TABLE]

Thus, $f_{v_{\inf}}({\bf x})=f_{v}({\bf x})$ and

[TABLE]

As $\phi$ is increasing, we get

[TABLE]

By uniqueness of minimizer, $f_{v_{\inf}}=f_{v}$. As a consequence, ${\bf v}_{\inf}$ defined by (4.21) satisfies (4.19). The proof is complete.   $\Box$

When ${\cal W}_{2}={\cal W}_{1}^{*}$, the above result can be simplified.

Corollary 4.8

Assume the same assumptions as in Theorem 4.4. If, in addition, ${\cal W}_{2}={\cal W}_{1}^{*}$ then the regularization network (4.20) possesses a unique solution $f_{{\bf v}_{\inf}}$ where ${\bf v}_{\inf}\in{\cal W}_{2}$ satisfies

[TABLE]

The above result covers representer theorems in existing RKBSs [43, 46, 48]. Especially, it covers the classical representer theorem for regularization networks in an RKHS.