Bochner integrals and neural networks

TL;DR

This paper develops a functional analytic framework for neural networks using Bochner integrals, establishing new theoretical foundations and properties of variation spaces as Banach spaces.

Contribution

It introduces a Bochner integral formula for neural networks and analyzes the structure of variation spaces within a functional analytic context.

Findings

Variation spaces are Banach spaces.

Established norm inequalities relating pointwise and Bochner integrals.

Derived a Bochner integral formula representing functions via weights and parametrized functions.

Abstract

A Bochner integral formula is derived that represents a function in terms of weights and a parametrized family of functions. Comparison is made to pointwise formulations, norm inequalities relating pointwise and Bochner integrals are established, variation-spaces and tensor products are studied, and examples are presented. The paper develops a functional analytic theory of neural networks and shows that variation spaces are Banach spaces.

Equations

$$b+rA:=\{b+ra|a\in A\}=\{y\in X:y=b+ra,a\in A\}.$$

$$\Vert \cdot {\Vert }_1=\Vert \cdot {\Vert }_{L^1(Y,\mu )}\text{mbox}and\Vert \cdot {\Vert }_{\infty }=\Vert \cdot {\Vert }_{L^{\infty }(Y,\mu ;\mathcal{X})};$$

$$s_{G,\mathcal{X}}:=\underset{g\in G}{\sup }\Vert g{\Vert }_{\mathcal{X}}<\infty .$$

$$B_{G,\mathcal{X}}:={\mathrm{cl}}_{\mathcal{X}}(\mathrm{conv}(\pm G)),\text{mbox}where\pm G:=G\cup -G.$$

$$\Vert f{\Vert }_{G,\mathcal{X}}:=\inf \{\lambda >0:f/\lambda \in B_{G,\mathcal{X}}\}.$$

$${\mathcal{X}}_G:=\{f\in \mathcal{X}:\exists \lambda >0\ni f/\lambda \in B_{G,\mathcal{X}}\}.$$

$$\Vert f{\Vert }_{\mathcal{X}}\le \Vert f{\Vert }_{G,\mathcal{X}}{s}_{G,\mathcal{X}}$$

$$B_{G,\mathcal{X}}={\mathrm{cl}}_{\mathcal{X}}(\mathrm{conv}(\pm G))\subseteq {\mathrm{cl}}_X(\mathrm{span}(G)),$$

$$B_{G,\mathcal{X}}=\left\{\sum _{n\ge 1}{c}_n{e}_n-\sum _{n\ge 1}{d}_n{e}_n:\forall n,c_n\ge 0,d_n\ge 0,\sum _{n\ge 1}(c_n+d_n)=1\right\}.$$

$$s=\sum _{j=1}^m\kappa (f_j){\chi }_{P_j},$$

$$I(s,\mu ):=\sum _{j=1}^m\mu (P_j)f_j\in \mathcal{X}.$$

$$\underset{k\to \infty }{\lim }\Vert s_k(y)-h(y){\Vert }_{\mathcal{X}}=0.$$

$$\underset{k\to \infty }{\lim }{\int }_Y\Vert s_k(y)-h(y){\Vert }_{\mathcal{X}}d\mu (y)=0.$$

$$I(h,\mu )\text{mbox}or\mathcal{B}-{\int }_{Y}h(y)d\mu (y).$$

$$h\in {\mathcal{L}}^1(Y,\mu ;\mathcal{X})\text{mbox}ifandonlyif{\int }_Y\Vert h(y){\Vert }_{\mathcal{X}}d\mu (y)<\infty .$$

$$\Vert I(h,\mu ){\Vert }_{\mathcal{X}}={\Vert \mathcal{B}-{\int }_{Y}h(y)d\mu (y)\Vert }_{\mathcal{X}}\le \Vert h{\Vert }_{L^1(Y,\mu ;\mathcal{X})}:={\int }_Y\Vert h(y){\Vert }_{X}d\mu (y).$$

$$T\left(\mathcal{B}-{\int }_{Y}h(y)d\nu (y)\right)=\mathcal{B}-{\int }_Y(T\circ h)(y)d\nu (y).$$

$$\mathcal{B}-{\int }_{Y}h(y)d\nu (y)\in \nu (Y){\mathrm{cl}}_X(\mathrm{conv}(\{\pm h(y):y\in Y\}).$$

$$\underset{y\in Y}{\sup }F(h(y))\ge {\int }_{Y}F(h(y))d\nu (y)=F(f)>\underset{y\in Y}{\sup }F(h(y)).$$

$$\underset{y\in Y\setminus N}{\sup }\Vert h(y){\Vert }_{\mathcal{X}}<\infty .$$

$$\Vert h{\Vert }_{L^{\infty }(Y,\nu ;X)}:=\inf \{B\ge 0:\exists \nu \text{mbox}-nullN\subset Y\ni \Vert h(y){\Vert }_X\le B,\forall y\in Y\setminus N\}.$$

$$w\Psi :y\mapsto w(y)\Psi (y),$$

$${\int }_Y\Vert h(y){\Vert }_{\mathcal{X}}d\nu (y)\le \nu (Y)\Vert h{\Vert }_{L^{\infty }(Y,\nu ;\mathcal{X})}.$$

$$\mathcal{B}-{\int }_{Y}w(y)\Phi (y)d\mu (y),\mathcal{B}-{\int }_Y\mathrm{sgn}(w(y))\Phi (y)d{\mu }_w(y)$$

$${\int }_Y\Vert [(\mathrm{sgn}\circ w)\Phi ](y){\Vert }_{\mathcal{X}}d{\mu }_w(y)={\int }_Y\Vert w(y)\Phi (y){\Vert }_{\mathcal{X}}d\mu (y),$$

$$F\left(\mathcal{B}-{\int }_{Y}w(y)\Phi (y)d\mu (y)\right)={\int }_{Y}F(w(y)\Phi (y))d\mu (y)$$

$$={\int }_{Y}w(y)F(\Phi (y))d\mu (y)={\int }_{Y}sgn(w(y))|w(y)|F(\Phi (y))d\mu (y)$$

$$={\int }_{Y}sgn(w(y))F(\Phi (y))d{\mu }_w(y)={\int }_{Y}F(sgn(w(y))\Phi (y))d{\mu }_w(y)$$

$$=F\left(\mathcal{B}-{\int }_{Y}sgn(w(y))\Phi (y)d{\mu }_w(y)\right).$$

$$f(x)={\int }_{Y}w(y)\varphi (x,y)d\mu (y)$$

$$\Vert f{\Vert }_{\mathcal{X}}\le \Vert w{\Vert }_{L^1(Y,\mu )}M.$$

$$f=\mathcal{B}-{\int }_Y(w\Phi )(y)d\mu (y).$$

Full text

Keywords: Variational norm, essentially bounded, strongly measurable, Bochner integration, tensor product, $L^{p}$ spaces, integral formula.

1 Introduction

A neural network utilizes data to find a function consistent with the data and with further “conceptual” data such as desired smoothness, boundedness, or integrability. The weights for a neural net and the functions embodied in the hidden units can be thought of as determining a finite sum that approximates some function. This finite sum is a kind of quadrature for an integral formula that would represent the function exactly.

This chapter uses abstract analysis to investigate neural networks. Our approach is one of enrichment: not only is summation replaced by integration, but also numbers are replaced by real-valued functions on an input set $\Omega$, the functions lying in a function space ${\mathcal{X}}$. The functions, in turn, are replaced by ${\mathcal{X}}$-valued measurable functions $\Phi$ on a measure space $Y$ of parameters. The goal is to understand approximation of functions by neural networks so that one can make effective choices of the parameters to produce a good approximation.

To achieve this, we utilize Bochner integration. The idea of applying this tool to neural nets is in Girosi and Anzellotti [14] and we developed it further in Kainen and Kůrková [23]. Bochner integrals are now being used in the theory of support vector machines and reproducing kernel Hilbert spaces; see the recent book by Steinwart and Christmann [42], which has an appendix of more than 80 pages of material on operator theory and Banach-space-valued integrals. Bochner integrals are also widely used in probability theory in connection with stochastic processes of martingale-type; see, e.g., [8, 39]. The corresponding functional analytic theory may help to bridge the gap between probabilistic questions and deterministic ones, and may be well-suited for issues that arise in approximation via neural nets.

Training to replicate given numerical data does not give a useful neural network for the same reason that parrots make poor conversationalists. The phenomenon of overfitting shows that achieving fidelity to data at all costs is not desirable; see, e.g., the discussion on interpolation in our other chapter in this book (Kainen, Kurková, and Sanguineti [45]). In approximation, we try to find a function close to the data that achieves desired criteria such as sufficient smoothness, decay at infinity, etc. Thus, a method of integration which produces functions in toto rather than numbers could be quite useful.

Enrichment has lately been utilized by applied mathematicians to perform image analysis and even to deduce global properties of sensor networks from local information. For instance, the Euler characteristic, ordinarily thought of as a discrete invariant, can be made into a variable of integration [7]. In the case of sensor networks, such an analysis can lead to effective computations in which theory determines a minimal set of sensors [40].

By modifying the traditional neural net focus on training sets of data so that we get to families of functions in a natural way, we aim to achieve methodological insight. Such a framework may lead to artificial neural networks capable of performing more sophisticated tasks.

The main result of this chapter is Theorem 12 which characterizes functions to be approximated in terms of pointwise integrals and Bochner integrals, and provides inequalities that relate corresponding norms. The relationship between integral formulas and neural networks has long been noted; e.g., [20, 6, 37, 13, 34, 29] We examine integral formulas in depth and extend their significance to a broader context.

An earlier version of the Main Theorem, including the bounds on variational norm by the $L^{1}$-norm of the weight function in a corresponding integral formula, was given in [23] and it also utilized functional (i.e., Bochner) integration. However, the version here is more general and further shows that if $\phi$ is a real-valued function on $\Omega\times Y$ (the cartesian product of input and parameter spaces), then the associated map $\Phi$ which maps the measure space to the Banach space defined by $\Phi(y)(x)=\phi(x,y)$ is measurable; cf. [42, Lemma 4.25, p. 125] where $\Phi$ is the “feature map.”

Other proof techniques are available for parts of the Main Theorem. In particular, Kurková [28] gave a different argument for part (iv) of the theorem, using a characterization of variation via peak functionals [31] as well as the theorem of Mazur (Theorem 13.1) used in the proof of Lemma 3.4. But the Bochner integral approach reveals some unexpected aspects of functional approximation which may be relevant for neural network applications.

Furthermore, the treatment of analysis and topology utilizes a number of basic theorems from the literature and provides an introduction to functional analysis motivated by its applicability. This is a case where neural nets provide a fresh perspective on classical mathematics. Indeed, theoretical results proved here were obtained in an attempt to better understand neural networks.

An outline of the paper is as follows: In section 2 we discuss variational norms; sections 3 and 4 present needed material on Bochner integrals. The Main Theorem (Theorem 12) on integral formulas is given in Section 5. In section 6 we show how to apply the Main Theorem to an integral formula for the Bessel potential function in terms of Gaussians. In section 7 we show how this leads to an inequality involving Gamma functions and provide an alternative proof by classical means. Section 8 interprets and extends the Main Theorem in the language of tensor products. Using tensor products, we replace individual ${\mathcal{X}}$-valued $\Phi$’s by families $\{\Phi_{j}:j\in J\}$ of such functions. This allows more nuanced representation of the function to be approximated. In section 9 we give a detailed example of concepts related to $G$-variation, while section 10 considers the relationship between pointwise integrals and evaluation of the corresponding Bochner integrals. Remarks on future directions are in section 11, and the chapter concludes with two appendices and references.

2 Variational norms and completeness

We assume that the reader has a reasonable acquaintance with functional analysis but have attempted to keep this chapter self-contained. Notations and basic definitions are given in Appendix I, while Appendix II has the precise statement of several important theorems from the literature which will be needed in our development.

Throughout this chapter, all linear spaces are over the reals ${\mathbb{R}}$. For $A$ any subset of a linear space $X$, $b\in X$, and $r\in{\mathbb{R}}$,

[TABLE]

Also, we sometimes use the abbreviated notation

[TABLE]

the standard notations on the right are explained in sections 12 and 4, resp. The symbol “$\ni$” stands for “such that.”

A set $G$ in a normed linear space ${\mathcal{X}}$ is fundamental (with respect to ${\mathcal{X}}$) if ${\rm cl}_{{\mathcal{X}}}\;({\rm span}\;G)={\mathcal{X}}$, where closure depends only on the topology induced by the norm. We call $G$ bounded with respect to ${\mathcal{X}}$ if

[TABLE]

We now review $G$-variation norms. These norms, which arise in connection with approximation of functions, were first considered by Barron [5], [6]. He treated a case where $G$ is a family of characteristic functions of sets satisfying a special condition. The general concept, formulated by Kůrková [24], has been developed in such papers as [30, 26, 27, 15, 16, 17].

Consider the set

[TABLE]

This is a symmetric, closed, convex subset of ${\mathcal{X}}$, with Minkowski functional

[TABLE]

The subset ${\mathcal{X}}_{G}$ of ${\mathcal{X}}$ on which this functional is finite is given by

[TABLE]

If $G$ is bounded, then $\|\cdot\|_{G,{\mathcal{X}}}$ is a norm on ${\mathcal{X}}_{G}$. In general ${\mathcal{X}}_{G}$ may be a proper subset of ${\mathcal{X}}$ even if $G$ is bounded and fundamental w.r.t. ${\mathcal{X}}$. See the example at the end of this section. The inclusion $\iota:{\mathcal{X}}_{G}\subseteq{\mathcal{X}}$ is linear and for every $f\in{\mathcal{X}}_{G}$

[TABLE]

Indeed, if $f/\lambda\in B_{G,{\mathcal{X}}}$, then $f/\lambda$ is a convex combination of elements of ${\mathcal{X}}$-norm at most $s_{G,{\mathcal{X}}}$, so $\|f\|_{{\mathcal{X}}}\leq\lambda\,s_{G,{\mathcal{X}}}$ establishing (3) by definition of variational norm. Hence, if $G$ is bounded in ${\mathcal{X}}$, the operator $\iota$ is bounded with operator norm not exceeding $s_{G,{\mathcal{X}}}$.

Proposition 2.1

*Let nonempty $G\subseteq{\mathcal{X}}$ a normed linear space. Then

(i) ${\rm span}\,G\subseteq{\mathcal{X}}_{G}\subseteq{\rm cl}_{\mathcal{X}}\,{\rm span}\,G$;

(ii) $G$ is fundamental if and only if ${\mathcal{X}}_{G}$ is dense in ${\mathcal{X}}$;

(iii) For $G$ bounded and ${\mathcal{X}}$ complete, $({\mathcal{X}}_{G},\|\cdot\|_{G,{\mathcal{X}}})$ is a Banach space.*

**Proof. ** (i) Let $f\in{\rm span}\,G$, then $f=\sum_{i=1}^{n}a_{i}g_{i}$, for real numbers $a_{i}$ and $g_{i}\in G$. We assume the $a_{i}$ are not all zero since [math] is in ${\mathcal{X}}_{G}$. Then $f=\lambda\sum_{i=1}^{n}|a_{i}|/\lambda(\pm g_{i})$, where $\lambda=\sum_{i=1}^{n}|a_{i}|$. Thus, $f$ is in $\lambda{\rm conv}(\pm G)\subseteq\lambda B_{G,{\mathcal{X}}}$. So $\|f\|_{G,{\mathcal{X}}}\leq\lambda$ and $f$ is in ${\mathcal{X}}_{G}$.

Likewise if $f$ is in ${\mathcal{X}}_{G}$, then for some $\lambda>0$, $f/\lambda$ is in

[TABLE]

so $f$ is in ${\rm cl}_{X}({\rm span}(G))$.

(ii) Suppose $G$ is fundamental. Then ${\mathcal{X}}=cl_{\mathcal{X}}({\rm span}\,G)={\rm cl}_{\mathcal{X}}({\mathcal{X}}_{G})$ by part (i). Conversely, if ${\mathcal{X}}_{G}$ is dense in ${\mathcal{X}}$, then ${\mathcal{X}}={\rm cl}_{\mathcal{X}}({\mathcal{X}}_{G})\subseteq{\rm cl}_{\mathcal{X}}({\rm span}\,G)\subseteq{\mathcal{X}}$, and $G$ is fundamental.

(iii) Let $\{f_{n}\}$ be a Cauchy sequence in ${\mathcal{X}}_{G}$. By (3) $\{f_{n}\}$ is a Cauchy sequence in ${\mathcal{X}}$ and has a limit $f$ in ${\mathcal{X}}$. The sequence $\|f_{n}\|_{G,{\mathcal{X}}}$ is bounded in ${\mathcal{X}}_{G}$, that is, there is a positive number M such that for all n $f_{n}/M\in B_{G,{\mathcal{X}}}$. Since $B_{G,{\mathcal{X}}}$ is closed in ${\mathcal{X}}$, $f/M$ is also in $B_{G,{\mathcal{X}}}$. Hence $\|f\|_{G,{\mathcal{X}}}\leq M$ and $f$ is in ${\mathcal{X}}_{G}$. Now given $\epsilon>0$ choose a positive integer $N$ such that $\|f_{n}-f_{k}\|_{G,{\mathcal{X}}}<\epsilon$ for $n,k\geq N$. In particular fix $n\geq N$, and consider a variable integer $k\geq N$. Then $\|f_{k}-f_{n}\|_{G,{\mathcal{X}}}<\epsilon$. So $(f_{k}-f_{n})/\epsilon\in B_{G,{\mathcal{X}}}$, and $f_{k}\in f_{n}+\epsilon B_{G,{\mathcal{X}}}$ for all $k\geq N$. But $f_{n}+\epsilon B_{G,{\mathcal{X}}}$ is closed in ${\mathcal{X}}$. Hence $f\in f_{n}+\epsilon B_{G,{\mathcal{X}}}$, and $\|f-f_{n}\|_{G,{\mathcal{X}}}\leq\epsilon$. So the sequence converges to $f$ in ${\mathcal{X}}_{G}$. $\Box$

The following example illustrates several of the above concepts. Take ${\mathcal{X}}$ to be a real separable Hilbert space with orthonormal basis $\{e_{n}:n=0,1,...\}$. Let $G=\{e_{n}:n=0,1,...\}$. Then

[TABLE]

Now $f\in{\mathcal{X}}$ is of the form $\sum_{n\geq 1}a_{n}e_{n}$ where $\|f\|_{{\mathcal{X}}}=\sqrt{\sum_{n\geq 1}a_{n}^{2}}$, and if $f\in{\mathcal{X}}_{G}$, then $a_{n}=\lambda(c_{n}-d_{n})$ for all $n$ and suitable $c_{n},d_{n}$. The minimal $\lambda$ can be obtained by taking $a_{n}=\lambda c_{n}$ when $a_{n}\geq 0$, and $a_{n}=-\lambda d_{n}$ when $a_{n}<0$. It then follows that $\|f\|_{G,{\mathcal{X}}}=\sum_{n\geq 1}|a_{n}|$. Hence when ${\mathcal{X}}$ is isomorphic to $\ell_{2}$, ${\mathcal{X}}_{G}$ is isomorphic to $\ell_{1}$. As $G$ is fundamental, by part(ii) above, the closure of $\ell_{1}$ in $\ell_{2}$ is $\ell_{2}$. This provides an example where ${\mathcal{X}}_{G}$ is not a closed subspace of ${\mathcal{X}}$ and so, while it is a Banach space w.r.t. the variational norm, it is not complete in the ambient-space norm.

3 Bochner integrals

The Bochner integral replaces numbers with functions and represents a broadranging extension, generalizing the Lebesgue integral from real-valued functions to functions with values in an arbitrary Banach space. Key definitions and theorems are summarized here for convenience, following the treatment in [44] (cf. [33]). Bochner integrals are used here (as in [23]) in order to prove a bound on variational norm.

Let $(Y,\mu)$ be a measure space. Let ${\mathcal{X}}$ be a Banach space with norm $\|\cdot\|_{{\mathcal{X}}}$. A function $s:Y\to{\mathcal{X}}$ is simple if it has a finite set of nonzero values $f_{j}\in{\mathcal{X}}$, each on a measurable subset $P_{j}$ of $Y$ with $\mu(P_{j})<\infty$, $1\leq j\leq m$, and the $P_{j}$ are pairwise-disjoint. Equivalently, a function $s$ is simple if it can be written in the following form:

[TABLE]

where $\kappa(f_{j}):Y\to{\mathcal{X}}$ denotes the constant function with value $f_{j}$ and $\chi_{P}$ denotes the characteristic function of a subset $P$ of $Y$. This decomposition is nonunique and we identify two functions if they agree $\mu$-almost everywhere - i.e., the subset of $Y$ on which they disagree has $\mu$-measure zero.

Define an ${\mathcal{X}}$-valued function $I$ on the simple functions by setting for $s$ of form (4)

[TABLE]

This is independent of the decomposition of $s$ [44, pp.130–132]. A function $h:Y\to{\mathcal{X}}$ is strongly measurable (w.r.t. $\mu$) if there exists a sequence $\{s_{k}\}$ of simple functions such that for $\mu$-a.e. $y\in Y$

[TABLE]

A function $h:Y\to{\mathcal{X}}$ is Bochner integrable (with respect to $\mu$) if it is strongly measurable and there exists a sequence $\{s_{k}\}$ of simple functions $s_{k}:Y\to{\mathcal{X}}$ such that

[TABLE]

If $h$ is strongly measurable and (5) holds, then the sequence $\{I(s_{k},\mu)\}$ is Cauchy and by completeness converges to an element in ${\mathcal{X}}$. This element, which is independent of the sequence of simple functions satisfying (5), is called the Bochner integral of $h$ (w.r.t. $\mu$) and denoted

[TABLE]

Let ${\mathcal{L}}^{1}(Y,\mu;{\mathcal{X}})$ denote the linear space of all strongly measurable functions from $Y$ to ${\mathcal{X}}$ which are Bochner integrable w.r.t. $\mu$; let $L^{1}(Y,\mu;{\mathcal{X}})$ be the corresponding set of equivalence classes (modulo $\mu$-a.e. equality). It is easily shown that equivalent functions have the same Bochner integral. Then the following elegant characterization holds.

Theorem 3.1** (Bochner)**

Let $({\mathcal{X}},\|\cdot\|_{{\mathcal{X}}})$ be a Banach space and $(Y,\mu)$ a measure space. Let $h:Y\to{\mathcal{X}}$ be strongly measurable. Then

[TABLE]

A consequence of this theorem is that $I:L^{1}(Y,\mu;{\mathcal{X}})\to{\mathcal{X}}$ is a continuous linear operator and

[TABLE]

In particular, the Bochner norm of $s$, $\|s\|_{L^{1}(Y,\mu;{\mathcal{X}})}$, is $\sum_{i}\mu(P_{i})\|g_{i}\|_{\mathcal{X}}$, where $s$ is a simple function satisfying (4).

For $Y$ a measure space and ${\mathcal{X}}$ a Banach space, $h:Y\to{\mathcal{X}}$ is weakly measurable if for every continuous linear functional $F$ on $X$ the composite real-valued function $F\circ h$ is measurable [43, pp. 130–134]. If $h$ is measurable, then it is weakly measurable since measurable followed by continuous is measurable: for $U$ open in ${\mathbb{R}}$, $(F\circ h)^{-1}(U)=h^{-1}(F^{-1}(U))$.

Recall that a topological space is separable if it has a countable dense subset. Let $\lambda$ denote Lebesgue measure on ${\mathbb{R}}^{d}$ and let $\Omega\subseteq{\mathbb{R}}^{d}$ be $\lambda$-measurable, $d\geq 1$. Then $L^{q}(\Omega,\lambda)$ is separable when $1\leq q<\infty$; e.g., [36, pp. 208]. A function $h:Y\to{\mathcal{X}}$ is $\mu$-almost separably valued ($\mu$-a.s.v.) if there exists a $\mu$-measurable subset $Y_{0}\subset Y$ with $\mu(Y_{0})=0$ and $h(Y\setminus Y_{0})$ is a separable subset of ${\mathcal{X}}$.

Theorem 3.2** (Pettis)**

Let $({\mathcal{X}},\|\cdot\|_{{\mathcal{X}}})$ be a Banach space and $(Y,\mu)$ a measure space. Suppose $h:Y\to{\mathcal{X}}$. Then $h$ is strongly measurable if and only if $h$ is weakly measurable and $\mu$-a.s.v.

The following basic result (see, e.g., [9]) was later extended by Hille to the more general class of closed operators. But we only need the result for bounded linear functionals, in which case the Bochner integral coincides with ordinary integration.

Theorem 3.3

Let $(Y,\nu)$ be a measure space, let ${\mathcal{X}}$, ${\mathcal{X}}^{\prime}$ be Banach spaces, and let $h\in{\mathcal{L}}^{1}(Y,\nu;{\mathcal{X}})$. If $T:{\mathcal{X}}\to{\mathcal{X}}^{\prime}$ is a bounded linear operator, then $T\circ h\in{\mathcal{L}}^{1}(Y,\nu;{\mathcal{X}}^{\prime})$ and

[TABLE]

There is a mean-value theorem for Bochner integrals (Diestel and Uhl [12, Lemma 8, p. 48]). We give their argument with a slightly clarified reference to the Hahn-Banach theorem.

Lemma 3.4

Let $(Y,\nu)$ be a finite measure space, let $X$ be a Banach space, and let $h:Y\to{\mathcal{X}}$ be Bochner integrable w.r.t. $\nu$. Then

[TABLE]

**Proof. ** Without loss of generality, $\nu(Y)=1$. Suppose $f:=I(h,\nu)\notin{\rm cl}_{X}({\rm conv}(\{\pm h(y):y\in Y\})$. By a consequence of the Hahn-Banach theorem given as Theorem 13.1 in Appendix II below), there is a continuous linear functional $F$ on $X$ such that $F(f)>\sup_{y\in Y}F(h(y))$. Hence, by Theorem 3.3,

[TABLE]

which is absurd. $\Box$

4 Spaces of Bochner integrable functions

In this section, we derive a few consequences of the results from the previous section which we shall need below.

A measurable function $h$ from a measure space $(Y,\nu)$ to a normed linear space ${\mathcal{X}}$ is called essentially bounded (w.r.t. $\nu$) if there exists a $\nu$-null set $N$ for which

[TABLE]

Let ${\mathcal{L}}^{\infty}(Y,\nu;{\mathcal{X}})$ denote the linear space of all strongly measurable, essentially bounded functions from $(Y,\nu)$ to ${\mathcal{X}}$. Let $L^{\infty}(Y,\nu;{\mathcal{X}})$ be its quotient space mod the relation of equality $\nu$-a.e. This is a Banach space with norm

[TABLE]

To simplify notation, we sometimes write $\|h\|_{\infty}$ for $\|h\|_{L^{\infty}(Y,\nu;X)}$ Note that if $\|h\|_{\infty}=c$, then $\|h(y)\|_{\mathcal{X}}\leq c$ for $\nu$-a.e. $y$. Indeed, for positive integers $k$, $\|h(y)\|_{\mathcal{X}}\leq c+(1/k)$ for $y$ not in a set of measure zero $N_{k}$ so $\|h(y\|_{\mathcal{X}}\leq c$ for $y$ not in the union $\bigcup_{k\geq 1}N_{k}$ also a set of measure zero.

We also have a useful fact whose proof is immediate.

Lemma 4.1

For every measure space $(Y,\mu)$ and Banach space ${\mathcal{X}}$, the natural map $\kappa_{\mathcal{X}}:{\mathcal{X}}\to L^{\infty}(Y,\mu;{\mathcal{X}})$ associating to each element $g\in{\mathcal{X}}$ the constant function from $Y$ to ${\mathcal{X}}$ given by $(\kappa_{\mathcal{X}}(g))(y)\equiv g$ for all $y$ in $Y$ is an isometric linear embedding.

Lemma 4.2

Let ${\mathcal{X}}$ be a separable Banach space, let $(Y,\mu)$ be a measure space, and let $w:Y\to{\mathbb{R}}$ and $\Psi:Y\to{\mathcal{X}}$ be $\mu$-measurable functions. Then $w\Psi$ is strongly measurable.

**Proof. ** By definition, $w\Psi$ is the function from $Y$ to ${\mathcal{X}}$ defined by

[TABLE]

where the multiplication is that of a Banach space element by a real number. Then $w\Psi$ is measurable because it is obtained from a pair of measurable functions by applying scalar multiplication which is continuous. Hence, by separability, Pettis’ Theorem 3.2, and the fact that measurable implies weakly measurable, we have strong measurability for $w\Psi$ (cf. [33, Lemma 10.3]). $\Box$

If $(Y,\nu)$ is a finite measure space, ${\mathcal{X}}$ is a Banach space, and $h:Y\to{\mathcal{X}}$ is strongly measurable and essentially bounded, then $h$ is Bochner integrable by Theorem 3.1. The following lemma, which follows from Lemma 4.2, allows us to weaken the hypothesis on the function by further constraining the space ${\mathcal{X}}$.

Lemma 4.3

Let $(Y,\nu)$ be a finite measure space, ${\mathcal{X}}$ a separable Banach space, and $h:Y\to{\mathcal{X}}$ be $\nu$-measurable and essentially bounded w.r.t. $\nu$. Then $h\in{\mathcal{L}}^{1}(Y,\nu;{\mathcal{X}})$ and

[TABLE]

Let $w\in{\mathcal{L}}^{1}(Y,\mu)$, and let $\mu_{w}$ be defined for $\mu$-measurable $S\subseteq Y$ by $\mu_{w}(S):=\int_{S}|w(y)|d\mu(y)$. For $t\neq 0$, ${\rm sgn}(t):=t/|t|$.

Theorem 4.4

Let $(Y,\mu)$ be a measure space, ${\mathcal{X}}$ a separable Banach space; let $w\in{\mathcal{L}}^{1}(Y,\mu)$ be nonzero $\mu$-a.e., let $\mu_{w}$ be the measure defined above, and let $\Phi:Y\to{\mathcal{X}}$ be $\mu$-measurable. If one of the Bochner integrals

[TABLE]

exists, then both exist and are equal.

**Proof. ** By Lemma 4.2, both $w\Phi$ and $({\rm sgn}\circ w)\Phi$ are strongly measurable. Hence, by Theorem 3.1, the respective Bochner integrals exist if and only if the ${\mathcal{X}}$-norms of the respective integrands have finite ordinary integral. But

[TABLE]

so the Bochner integral $I(({\rm sgn}\circ w)\Phi,\mu_{w})$ exists exactly when $I(w\Phi,\mu)$ does. Further, the respective Bochner integrals are equal since for any continuous linear functional $F$ in ${\mathcal{X}}^{*}$, by Theorem 3.3

[TABLE]

$\Box$

Corollary 4.5

Let $(Y,\mu)$ be a $\sigma$-finite measure space, ${\mathcal{X}}$ a separable Banach space, $w:Y\to{\mathbb{R}}$ be in ${\mathcal{L}}^{1}(Y,\mu)$ and $\Phi:Y\to{\mathcal{X}}$ be in ${\mathcal{L}}^{\infty}(Y,\mu;{\mathcal{X}})$. Then $w\Phi$ is Bochner integrable w.r.t. $\mu$.

**Proof. ** By Lemma 4.2, $({\rm sgn}\circ w)\Phi$ is strongly measurable, and Lemma 4.3 then implies that the Bochner integral $I(({\rm sgn}\circ w)\Phi,\mu_{w})$ exists since $\mu_{w}(Y)=\|w\|_{L^{1}(Y,\mu)}<\infty$. So $w\Phi$ is Bochner integrable by Theorem 4.4. $\Box$

5 Main theorem

In the next result, we show that certain types of integrands yield integral formulas for functions $f$ in a Banach space of $L^{p}$-type both pointwise and at the level of Bochner integrals. Furthermore, the variational norm of $f$ is shown to be bounded by the $L^{1}$-norm of the weight function from the integral formula. Equations (9) and (10) and part (iv) of this theorem were derived in a similar fashion by one of us with Kůrková in [23] under more stringent hypotheses; see also [13, eq. (12)].

Theorem 5.1

*Let $(\Omega,\rho)$, $(Y,\mu)$ be $\sigma$-finite measure spaces, let $w$ be in ${\mathcal{L}}^{1}(Y,\mu)$, let ${\mathcal{X}}=L^{q}(\Omega,\rho)$, $q\in[1,\infty)$, be separable, let $\phi:\Omega\times Y\to{\mathbb{R}}$ be $\rho\times\mu$-measurable, let $\Phi:Y\to{\mathcal{X}}$ be defined for each $y$ in $Y$ by $\Phi(y)(x):=\phi(x,y)$ for $\rho$-a.e. $x\in\Omega$ and suppose that for some $M<\infty$, $\|\Phi(y)\|_{\mathcal{X}}\leq M$ for $\mu$-a.e. $y$. Then the following hold:

*(i) For $\rho$-a.e. $x\in\Omega$, the integral $\int_{Y}w(y)\phi(x,y)d\mu(y)$ exists and is finite.

(ii) The function $f$ defined by

[TABLE]

is in ${\mathcal{L}}^{q}(\Omega,\rho)$ and its equivalence class, also denoted by $f$, is in $L^{q}(\Omega,\rho)={\mathcal{X}}$ and satisfies

[TABLE]

(iii) The function $\Phi$ is measurable and hence in ${\mathcal{L}}^{\infty}(Y,\mu;{\mathcal{X}})$, and $f$ is the Bochner integral of $w\Phi$ w.r.t. $\mu$, i.e.,

[TABLE]

(iv) For $G=\{\Phi(y):\|\Phi(y)\|_{\mathcal{X}}\leq\|\Phi\|_{L^{\infty}(Y,\mu;{\mathcal{X}})}\}$, $f$ is in ${\mathcal{X}}_{G}$, and

[TABLE]

and as in (1)

[TABLE]

**Proof. ** (i) Consider the function $(x,y)\longmapsto|w(y)||\phi(x,y)|^{q}$. This is a well-defined $\rho\times\mu$-measurable function on $\Omega\times Y$. Furthermore its repeated integral

[TABLE]

exists and is bounded by $\|w\|_{1}M^{q}$ since $\Phi(y)\in L^{q}(\Omega,\rho)$ and $\|\Phi(y)\|_{q}^{q}\leq M^{q}$ for a. e. y. and $w\in L^{1}(Y,\mu)$. By Fubini’s Theorem 13.2 the function $y\longmapsto|w(y)||\phi(x,y)|^{q}$ is in $L^{1}(Y,\mu)$ for a.e. x. But the inequality

[TABLE]

shows that the function $y\longmapsto|w(y)||\phi(x,y)|$ is dominated by the sum of two integrable functions. Hence the integrand in the definition of $f(x)$ is integrable for a. e. x, and $f$ is well-defined almost everywhere.

(ii) The function $G(u)=u^{q}$ is a convex function for $u\geq 0$. Accordingly by Jensen’s inequality (Theorem 13.3 below),

[TABLE]

provided both integrals exist and $\sigma$ is a probability measure on the measurable space $Y$. We take $\sigma$ to be defined by the familiar formula:

[TABLE]

for $\mu$-measurable sets A in Y, so that integration with respect to $\sigma$ reduces to a scale factor times integration of $|w(y|d\mu(y)$. Since we have established that both $|w(y)||\phi(x,y)|$ and $|w(y)||\phi(x,y)|^{q}$ are integrable with respect to $\mu$ for a.e. x, we obtain:

[TABLE]

[TABLE]

for a.e. x. But we can now integrate both side with respect to $d\rho(x)$ over $\Omega$ because of the integrability noted above in connection with Fubini’s Theorem. Thus $f\in{\mathcal{X}}=L^{q}(\Omega,\rho)$ and $\|f\|^{q}_{\mathcal{X}}\leq\|w\|_{1}^{q}\,M^{q}$, again interchanging order.

(iii) First we show that $\Phi^{-1}$ of the open ball centered at $g$ of radius ${\varepsilon}$, $B(g,{\varepsilon}):=\{y:\|\Phi(y)-g\|_{\mathcal{X}}<{\varepsilon}\}$, is a $\mu$-measurable subset of $Y$ for each $g$ in ${\mathcal{X}}$ and ${\varepsilon}>0$. Note that

[TABLE]

for all $y$ in $Y$ where $x\mapsto\phi(x,y)$ and $x\mapsto g(x)$ are $\rho$-measurable functions representing the elements $\Phi(y)$ and $g$ belonging to ${\mathcal{X}}=L^{q}(Y,\mu)$. Since $(Y,\mu)$ is $\sigma$-finite, we can find a strictly positive function $w_{0}$ in ${\mathcal{L}}^{1}(Y,\mu)$. (For example, let $w_{0}=\sum_{n\geq 1}(1/n^{2})\chi_{Y_{n}}$, where $\{Y_{n}:n\geq 1\}$ is a countable disjoint partition of $Y$ into $\mu$-measurable sets of finite measure.) Then $w_{0}(y)|\phi(x,y)-g(x)|^{q}$ is a $\rho\times\mu$-measurable function on $\Omega\times Y$, and

[TABLE]

By Fubini’s Theorem 13.2, $y\mapsto w_{0}(y)\|\Phi(y)-g\|_{\mathcal{X}}^{q}$ is $\mu$-measurable. Since $w_{0}$ is $\mu$-measurable and strictly positive, $y\mapsto\|\Phi(y)-g\|_{\mathcal{X}}^{q}$ is also $\mu$-measurable and so $B(g,{\varepsilon})$ is measurable. Hence, $\Phi:Y\to{\mathcal{X}}$ is measurable. Thus, $\Phi$ is essentially bounded, with essential sup $\|\Phi\|_{L^{\infty}(Y,\mu;{\mathcal{X}})}\leq M$. (In (9), $M$ can be replaced by this essential sup.)

By Corollary 4.5, $w\Phi$ is Bochner integrable. To prove that $f$ is the Bochner integral, using Theorem 4.4, we show that for each bounded linear functional $F\in{\mathcal{X}}^{*}$, $F(I({\rm sgn}\circ w\Phi,\mu_{w}))=F(f)$. By the Riesz representation theorem [35, p. 316], for any such $F$ there exists a (unique) $g_{F}\in{\mathcal{L}}^{p}(\Omega,\rho)$, $p=1/(1-q^{-1})$, such that for all $g\in{\mathcal{L}}^{q}(\Omega,\rho)$, $F(g)=\int_{\Omega}g_{F}(x)g(x)d\rho(x)$. By Theorem 3.3,

[TABLE]

But for $y\in Y$, $F({\rm sgn}(w(y))\Phi(y))={\rm sgn}(w(y))F(\Phi(y)$, so

[TABLE]

Also, using (8),

[TABLE]

The integrand of the iterated integrals is measurable with respect to the product measure $\rho\times\mu$, so by Fubini’s Theorem the iterated integrals are equal provided that one of the corresponding absolute integrals is finite. Indeed,

[TABLE]

By Hölder’s inequality, for every $y$,

[TABLE]

using the fact that ${\mathcal{X}}=L^{q}(\Omega,\rho)$. Therefore, by the essential boundedness of $\Phi$ w.r.t. $\mu$, the integrals in (13) are at most

[TABLE]

Hence, $f$ is the Bochner integral of $w\Phi$ w.r.t. $\mu$.

(iv) We again use Lemma 3.4. Let $Y_{0}$ be a measurable subset of $Y$ with $\mu(Y_{0})=0$ and for $Y^{\prime}=Y\setminus Y_{0}$, $\Phi(Y^{\prime})=G$; see the remark following the definition of essential supremum. But restricting ${\rm sgn}\circ w$ and $\Phi$ to $Y^{\prime}$, one has

[TABLE]

hence, $f\in\mu_{w}(Y){\rm cl}_{\mathcal{X}}{\rm conv}(\pm G)$. Thus, $\|w\|_{L^{1}(Y,\mu)}=\mu_{w}(Y)\geq\|f\|_{G,{\mathcal{X}}}$. $\Box$

6 An example involving the Bessel potential

Here we review an example related to the Bessel functions which was considered in [21] for $q=2$. In the following section, this Bessel-potential example is used to find an inequality related to the Gamma function.

Let ${\mathcal{F}}$ denote the Fourier transform, given for $f\in L^{1}({\mathbb{R}}^{d},\lambda)$ and $s\in{\mathbb{R}}^{d}$ by

[TABLE]

where $\lambda$ is Lebesgue measure and $dx$ means $d\lambda(x)$. For $r>0$, let

[TABLE]

Since the Fourier transform is an isometry of ${\mathcal{L}}^{2}$ onto itself (Parseval’s identity), and ${\hat{\beta_{r}}}$ is in ${\mathcal{L}}^{2}({\mathbb{R}}^{d})$ for $r>d/2$ (which we now assume), there is a unique function $\beta_{r}$, called the Bessel potential of order $r$, having ${\hat{\beta_{r}}}$ as its Fourier transform. See, e.g., [2, p. 252]. If $1\leq q<\infty$ and $r>d/q$, then ${\hat{\beta_{r}}}\in{\mathcal{L}}^{q}({\mathbb{R}}^{d})$ and

[TABLE]

Indeed, by radial symmetry, $(\|{\hat{\beta_{r}}}\|_{{\mathcal{L}}^{q}})^{q}=\int_{{\mathbb{R}}^{d}}(1+\|x\|^{2})^{-qr/2}dx=\omega_{d}I$, where $I=\int_{0}^{\infty}(1+\rho^{2})^{-qr/2}\rho^{d-1}d\rho$ and $\omega_{d}:=2\pi^{d/2}/\Gamma(d/2)$ is the area of the unit sphere in ${\mathbb{R}}^{d}$ [11, p. 303]. Substituting $\sigma=\rho^{2}$ and $d\rho=(1/2)\sigma^{-1/2}d\sigma$, and using [10, p. 60], we find that

[TABLE]

establishing (14).

For $b>0$, let $\gamma_{b}:{\mathbb{R}}^{d}\to{\mathbb{R}}$ denote the scaled Gaussian $\gamma_{b}(x)=e^{-b\|x\|^{2}}$. A simple calculation shows that the $L^{q}$-norm of $\gamma_{b}$:

[TABLE]

Indeed, using $\int_{-\infty}^{\infty}\exp(-t^{2})dt=\pi^{1/2}$, we obtain:

[TABLE]

We now express the Bessel potential as an integral combination of Gaussians. The Gaussians are normalized in $L^{q}$ and the corresponding weight function $w$ is explicitly given. The integral formula is similar to one in Stein [41]. By our main theorem, this is an example of (8) and can be interpreted either as a pointwise integral or as a Bochner integral.

Proposition 6.1

For $d$ a positive integer, $q\in[1,\infty)$, $r>d/q$, and $s\in{\mathbb{R}}^{d}$

[TABLE]

where

[TABLE]

and

[TABLE]

**Proof. ** Let

[TABLE]

Putting $u=t(1+\|s\|^{2})$ and $dt=du(1+\|s\|^{2})^{-1}$, we obtain

[TABLE]

Using the norm of the Gaussian (15), we arrive at

[TABLE]

which is the result desired. $\Box$

Now we apply Theorem 12 with $Y=(0,\infty)$ and $\phi(s,t)=\gamma_{t}^{o}(s)=\gamma_{t}(s)/\|\gamma_{t}\|_{L^{q}({\mathbb{R}}^{d})}$ to bound the variational norm of $\hat{\beta_{r}}$ by the $L^{1}$-norm of the weight function.

Proposition 6.2

For $d$ a positive integer, $q\in[1,\infty)$, and $r>d/q$,

[TABLE]

where $G=\{\gamma_{t}^{o}:0<t<\infty\}$ and ${\mathcal{X}}={\mathcal{L}}^{q}({\mathbb{R}}^{d})$.

**Proof. ** By (11) and Proposition 6.1, we have

[TABLE]

where $k=(\pi/q)^{d/2q}/\Gamma(r/2)$, and by definition, the integral is $\Gamma(r/2-d/2q)$. $\Box$

7 Application: A Gamma function inequality

The inequalities among the variational norm $\|\cdot\|_{G,{\mathcal{X}}}$, the Banach space norm $\|\cdot\|_{\mathcal{X}}$, and the $L^{1}$-norm of the weight function, established in the Main Theorem, allow us to derive other inequalities. The Bessel potential $\beta_{r}$ of order $r$ considered above provides an example.

Let $d$ be a positive integer, $q\in[1,\infty)$, and $r>d/q$. By Proposition 6.2 and (14) of the last section, and by (12) of the Main Theorem, we have

[TABLE]

Hence, with $a=r/2-d/2q\,$ and $s=r/2$, this becomes

[TABLE]

In fact, (17) holds if $s,a,d,q$ satisfy (i) $s>a>0$ and (ii) $s-a=d/2q$ for some $d\in Z^{+}$ and $q\in[1,\infty)$. As $a>0$, $r>d/q$. If $T=\{t>0:t=d/2q\;$ for some $\;d\in Z^{+},q\in[1,\infty)\}$, then $T=(0,\frac{1}{2}]\cup(0,1]\cup(0,\frac{3}{2}]\cup\ldots=(0,\infty)$, so there always exist $d$, $q$ satisfying (ii); the smallest such $d$ is $\lceil 2(s-a)\rceil$.

The inequality (17) suggests that the Main Theorem can be used to establish other inequalities of interest among classical functions. We now give a direct argument for the inequality. Its independent proof confirms our function-theoretic methods and provides additional generalization.

We begin by noting that in (17) it suffices to take $d=2q(s-a)$. If the inequality is true in that case, it is true for all real numbers $d\leq 2q(s-a)$. Thus, we wish to establish that

[TABLE]

is a strictly increasing function of $s$ for $q>1$ and $s>0$. (For $q=1$ this function is constant.)

Equivalently, we show that

[TABLE]

is a strictly increasing function of $s$ for $q>1$ and $s>0$.

Differentiating with respect to $s$, we obtain:

[TABLE]

where $\psi$ is the digamma function. It suffices to establish that $A_{s}(q)>0$ for $q>1$, $s>0$. Note that $A_{s}(1)=0$. Now consider

[TABLE]

This derivative is positive if and only if $\psi^{\prime}(qs)>\frac{1}{qs}$ for $q>1$, $s>0$.

It remains to show that $\psi^{\prime}(x)>\frac{1}{x}$ for $x>0$. Using the power series for $\psi^{\prime}$ [1, 6.4.10], we have for $x>0$,

[TABLE]

8 Tensor-product interpretation

The basic paradigm of feedforward neural nets is to select a single type of computational unit and then build a network based on this single type through a choice of controlling internal and external parameters so that the resulting network function approximates the target function; see [45]. However, a single type of hidden unit may not be as effective as one based on a plurality of hidden-unit types. Here we explore a tensor-product interpretation which may facilitate such a change in perspective.

Long ago Hille and Phillips [19, p. 86] observed that the Banach space of Bochner integrable functions from a measure space $(Y,\mu)$ into a Banach space ${\mathcal{X}}$ has a fundamental set consisting of two-valued functions, achieving a single non-zero value on a measurable set of finite measure. Indeed, every Bochner integrable function is a limit of simple functions, and each simple function (with a finite set of values achieved on disjoint parts $P_{j}$ of the partition) can be written as a sum of characteristic functions, weighted by members of the Banach space. If $s$ is such a simple function, then

[TABLE]

where the $\chi_{j}$ are the characteristic functions of the $P_{j}$ and the $g_{j}$ are in ${\mathcal{X}}$. (If, for example, $Y$ is embedded in a finite-dimensional Euclidean space, the partition could consist of generalized rectangles.)

Hence, if $f={\mathcal{B}}-\int_{Y}h(y)d\mu(y)$ is the Bochner integral of $h$ with respect to some measure $\mu$, then $f$ can be approximated as closely as desired by elements in ${\mathcal{X}}$ of the form

[TABLE]

where $Y=\bigcup_{i=1}^{n}P_{i}$ is a $\mu$-measurable partition of $Y$.

Note that given a $\sigma$-finite measure space $(Y,\mu)$ and a separable Banach space ${\mathcal{X}}$, every element $f$ in ${\mathcal{X}}$ is (trivially) the Bochner integral of any integrand $w\cdot\kappa(f)$, where $w$ is a nonnegative function on $Y$ with $\|w\|_{L^{1}(Y,\mu)}=1$ (see part (iii) of Theorem 12) and $\kappa(f)$ denotes the constant function on $Y$ with value $f$. In effect, $f$ is in ${\mathcal{X}}_{G}$ when $G=\{f\}$. When $\Phi$ is chosen first (or more precisely $\phi$ as in our Main Theorem), then $f$ may or may not be in ${\mathcal{X}}_{G}$. According to the Main Theorem, $f$ is in ${\mathcal{X}}_{G}$ when it is given by an integral formula involving $\Phi$ and some $L^{1}$ weight function. In this case, $G=\Phi(Y)\cap B$ where $B$ is the ball in ${\mathcal{X}}$ of radius $\|\Phi\|_{L^{\infty}(Y,\mu;{\mathcal{X}})}$.

In general, the elements $\Phi(y),\;y\in Y$ of the Banach space involved in some particular approximation for $f$ will be distinct functions of some general type obtained by varying the parameter $y$. For instance, kernels, radial basis functions, perceptrons, or various other classes of computational units can be used, and when these computational-unit-classes determine fundamental sets, by Proposition 2.1, it is possible to obtain arbitrarily good approximations. However, Theorem 8.1 below suggests that having a finite set of distinct types $\Phi_{i}:Y\to{\mathcal{X}}$ may allow a smaller “cost” for approximation, if we regard

[TABLE]

as the cost of the approximation

[TABLE]

We give a brief sketch of the ideas, following Light and Cheney [33].

Let ${\mathcal{X}}$ and ${\mathcal{Z}}$ be Banach spaces. Let ${\mathcal{X}}\otimes{\mathcal{Z}}$ denote the linear space of equivalence classes of formal expressions

[TABLE]

where these expressions are equivalent if for every $F\in{\mathcal{X}}^{*}$

[TABLE]

that is, if the associated operators from ${\mathcal{X}}^{*}\to{\mathcal{Z}}$ are identical, where ${\mathcal{X}}^{*}$ is the algebraic dual of ${\mathcal{X}}$. The resulting linear space ${\mathcal{X}}\otimes{\mathcal{Z}}$ is called the algebraic tensor product of ${\mathcal{X}}$ and ${\mathcal{Z}}$. We can extend ${\mathcal{X}}\otimes{\mathcal{Z}}$ to a Banach space by completing it with respect to a suitable norm. Consider the norm defined for $t\in{\mathcal{X}}\otimes{\mathcal{Z}}$,

[TABLE]

and complete the algebraic tensor product with respect to this norm; the result is denoted ${\mathcal{X}}\otimes_{\gamma}{\mathcal{Z}}$.

In [33, Thm. 1.15, p. 11], Light and Cheney showed that for any measure space $(Y,\mu)$ and any Banach space ${\mathcal{X}}$ the linear map

[TABLE]

given by

[TABLE]

is well-defined and extends to a map

[TABLE]

which is an isometric isomorphism of the completed tensor product onto the space $L^{1}(Y,\mu;X)$ of Bochner-integrable functions.

The following theorem extends the function $\Lambda_{\mathcal{X}}$ via the natural embedding $\kappa_{\mathcal{X}}$ of ${\mathcal{X}}$ into the space of essentially bounded ${\mathcal{X}}$-valued functions defined in section 4.

Theorem 8.1

Let ${\mathcal{X}}$ be a separable Banach space and let $(Y,\mu)$ be a $\sigma$-finite measure space. Then there exists a continuous linear surjection

[TABLE]

Furthermore, $e$ makes the following diagram commutative:

[TABLE]

where the two horizontal arrows $a$ and $d$ are the isometric isomorphisms $\Lambda_{\mathcal{X}}^{\gamma}$ and $\Lambda_{L^{\infty}(Y,\mu;{\mathcal{X}})}^{\gamma}$; the left-hand vertical arrow $b$ is induced by $1\otimes\kappa_{\mathcal{X}}$, while the right-hand vertical arrow $c$ is induced by post-composition with $\kappa_{\mathcal{X}}$, i.e., for any $h$ in $L^{1}(Y,\mu;{\mathcal{X}})$,

[TABLE]

**Proof. ** The map

[TABLE]

defines a linear function $L^{1}(Y,\mu)\otimes L^{\infty}(Y,\mu;{\mathcal{X}})\to L^{1}(Y,\mu;{\mathcal{X}})$; indeed, it takes values in the Bochner integrable functions as by our Main Theorem each summand is in the class.

To see that $e^{\prime}$ extends to $e$ on the $\gamma$-completion,

[TABLE]

[TABLE]

[TABLE]

Hence, $\|e(t)\|_{L^{1}(Y,\mu;{\mathcal{X}})}\leq\gamma(t)$, so the map $e$ is continuous. $\Box$

9 An example involving bounded variation on an interval

The following example, more elaborate than the one following Proposition 2.1, is treated in part by Barron [6] and Kurková [25].

Let ${\mathcal{X}}$ be the set of equivalence classes of (essentially) bounded Lebesgue-measurable functions on $[a,b]$, $a,b<\infty$, i.e., ${\mathcal{X}}=L^{\infty}([a,b])$, with norm $\|f\|_{\mathcal{X}}:=\inf\{M:|f(x)|\leq M\;\mbox{for almost every}\;x\in[a,b]\}$. Let $G$ be the set of equivalence classes of all characteristic functions of closed intervals of the forms $[a,b]$, or $[a,c]$ or $[c,b]$ with $a<c<b$. These functions are the restrictions of characteristic functions of closed half-lines to $[a,b]$. The equivalence relation is $f\sim g$ if and only if $f(x)=g(x)$ for almost every $x$ in $[a,b]$ (with respect to Lebesgue measure).

Let $BV([a,b])$ be the set of all equivalence classes of functions on $[a,b]$ with bounded variation; that is, each equivalence class contains a function $f$ such that the total variation $V(f,[a,b])$ is finite, where total variation is the largest possible total movement of a discrete point which makes a finite number of stops as x varies from $a$ to $b$, maximized over all possible ways to choose a finite list of intermediate points, that is,

[TABLE]

In fact, each equivalence class $[f]$ contains exactly one function $f^{*}$ of bounded variation that satisfies the continuity conditions:

(i) $f^{*}$ is right-continuous at $c$ for $c\in[a,b)$, and

(ii) $f^{*}$ is left-continuous at $b$.

Moreover, $V(f^{*},[a,b])\leq V(f,[a,b])$ for all $f\sim f^{*}$.

To see this, recall that every function $f$ of bounded variation is the difference of two nondecreasing functions $f=f_{1}-f_{2}$, and $f_{1},f_{2}$ are necessarily right-continuous except at a countable set. We can take $f_{1}(x):=V(f,[a,x])+K$, where $K$ is an arbitrary constant, and $f_{2}(x):=V(f,[a,x])+K-f(x)$ for $x\in[a,b]$. Now redefine both $f_{1}$ and $f_{2}$ at countable sets to form $f_{1}^{*}$ and $f_{2}^{*}$ which satisfy the continuity conditions and are still nondecreasing on $[a,b]$. Then $f^{*}:=f_{1}^{*}-f_{2}^{*}$ also satisfies the continuity conditions. It is easily shown that $V(f^{*},[a,b])\leq V(f,[a,b]).$ Since any equivalence class in ${\mathcal{X}}$ can contain at most one function satisfying (i) and (ii) above, it follows that $f^{*}$ is unique and that $V(f^{*},[a,b])$ minimizes the total variation for all functions in the equivalence class. Recall that $\chi_{[a,b]}=\chi([a,b])$ denotes the characteristic function of the interval $[a,b]$, etc.

Proposition 9.1

*Let ${\mathcal{X}}=L^{\infty}([a,b])$ and let $G$ be the subset of characteristic functions

$\chi([a,b]),\chi([a,c]),\chi([c,b]),a<c<b$ (up to sets of Lebesgue-measure zero).

Then ${\mathcal{X}}_{G}=BV([a,b])$, and*

[TABLE]

where $f^{*}$ is the member of $[f]$ satisfying the continuity conditions (i) and (ii).

**Proof. ** Let $C_{G,{\mathcal{X}}}$ be the set of equivalence classes of functions of the form

[TABLE]

where $k$ is a positive integer, $q,r\geq 0$, for $1\leq n\leq k$, $s_{n},t_{n},u_{n},v_{n}\geq 0,\;$ and

[TABLE]

All of the functions so exhibited have bounded variation $\leq 1$ and hence $C_{G,{\mathcal{X}}}\subseteq BV([a,b])$.

We will prove that a sequence in $C_{G,{\mathcal{X}}}$ converges in ${\mathcal{X}}$-norm to a member of $BV([a,b])$ and this will establish that $B_{G,{\mathcal{X}}}$ is a subset of $BV([a,b])$ and hence that ${\mathcal{X}}_{G}$ is a subset of $BV([a,b])$.

Let $\{[f_{k}]\}$ be a sequence in $C_{G,{\mathcal{X}}}$ that is Cauchy in the ${\mathcal{X}}$-norm. Without loss of generality, we pass to the sequence $\{f^{*}_{k}\}$, which is Cauchy in the sup-norm since $x\mapsto|f^{*}_{k}(x)-f^{*}_{j}(x)|$ satisfies the continuity conditions (i) and (ii). Thus, $\{f^{*}_{k}\}$ converges pointwise-uniformly and in the sup-norm to a function $f$ on $[a,b]$ also satisfying (i) and (ii) with finite sup-norm and whose equivalence class has finite ${\mathcal{X}}$-norm.

Let $\{x_{1},\ldots,x_{n}\}$ satisfy $a\leq x_{1}<x_{2}<\cdots<x_{n}\leq b$. Then

[TABLE]

for every $k$ , where par abus de notation $f_{k}$ denotes the member of $[f_{k}]$ satisfying (20). Letting $k$ tend to infinity and then varying $n$ and $x_{1},\ldots,x_{n}$, we obtain $V(f,[a,b])\leq 1$ and so $[f]\in BV([a,b])$.

It remains to show that everything in $BV([a,b])$ is actually in ${\mathcal{X}}_{G}$. Let $g$ be a nonnegative nondecreasing function on $[a,b]$ satisfying the continuity conditions (i) and (ii) above. Given a positive integer $n$, there exists a positive integer $m\geq 2$ and $a=a_{1}<a_{2}<\cdots<a_{m}=b$ such that $g(a_{i+1}^{-})-g(a_{i})\leq 1/n$ for $i=1,\ldots,m-1$. Indeed, for $2\leq i\leq m-1$, let $a_{i}:=\min\{x|g(a)+\frac{(i-1)}{n}\leq g(x)\}$. (Moreover, it follows that the set of $a_{i}$’s include all points of left-discontinuity of $g$ such that the jump $g(a_{i})-g(a_{i}^{-})$ is greater than $1/n$.) Let $g_{n}:[a,b]\to{\mathbb{R}}$ be defined as follows:

[TABLE]

[TABLE]

[TABLE]

Then $[g_{n}]$ belongs to $g(a_{m-1})C_{G,{\mathcal{X}}}$, and a fortiori to $g(b)C_{G,{\mathcal{X}}}$ as well as of ${\mathcal{X}}_{G}$, and $\|[g_{n}]\|_{G,{\mathcal{X}}}\leq g(b)$. Moreover, $\|[g_{n}]-[g]\|_{\mathcal{X}}\leq 1/n$. Therefore, since $B_{G,{\mathcal{X}}}={\rm cl}_{\mathcal{X}}(C_{G,{\mathcal{X}}})$, $[g]$ is in $g(b)B_{G,{\mathcal{X}}}$ and accordingly $[g]$ is in ${\mathcal{X}}_{G}$ and

[TABLE]

Let $[f]$ be in $BV([a,b])$ and let $f^{*}=f^{*}_{1}-f^{*}_{2}$, as defined above, for this purpose we take $K=|f^{*}(a)|$. This guarantees that both $f^{*}_{1}$ and $f^{*}_{2}$ are nonnegative. Accordingly, $[f]=[f^{*}_{1}]-[f^{*}_{2}]$, and is in ${\mathcal{X}}_{G}$. Furthermore, $\|[f]\|_{G,{\mathcal{X}}}\leq\|[f^{*}_{1}]\|_{G,{\mathcal{X}}}+\|[f^{*}_{2}]\|_{G,{\mathcal{X}}}\leq f^{*}_{1}(b)+f^{*}_{2}(b)=V(f^{*},[a,b])+|f^{*}(a)|+V(f^{*},[a,b])+|f^{*}(a)|-f^{*}(b)\leq 2V(f^{*},[a,b])+|f^{*}(a)|$. The last inequality follows from the fact that $V(f^{*},[a,b])+|f^{*}(a)|-f^{*}(b)\geq|f^{*}(b)-f^{*}(a)|+|f^{*}(a)|-f^{*}(b)\geq 0$. $\Box$

An argument similar to the above shows that $BV([a,b])$ is a Banach space under the norm $2V(f^{*},[a,b])+|f^{*}(a)|$ (with or without the $2$). The identity map from $BV([a,b])$ (with this norm) to $({\mathcal{X}}_{G},\|\cdot\|_{G,{\mathcal{X}}}$, is continuous (by Proposition 9.1) and it is also onto. Accordingly, by the Open Mapping Theorem (e.g., Yosida [43, p. 75]) the map is open, hence a homeomorphism, so the norms are equivalent. Thus, in this example, ${\mathcal{X}}_{G}$ is a Banach space under these two equivalent norms.

Note however that the ${\mathcal{X}}$-norm restricted to ${\mathcal{X}}_{G}$ does not give a Banach space structure; i.e., ${\mathcal{X}}_{G}$ is not complete in the ${\mathcal{X}}$-norm. Indeed, with ${\mathcal{X}}=L^{\infty}([0,1])$. Let $f_{n}$ be $1/n$ times the characteristic function of the disjoint union of $n^{2}$ closed intervals contained within the unit interval. Then $\|[f_{n}]\|_{\mathcal{X}}=1/n$ but $\|[f_{n}]\|_{G,{\mathcal{X}}}\geq Cn$, some $C>0$, since the $\|\cdot\|_{G,{\mathcal{X}}}$ is equivalent to the total-variation norm. While $\{f_{n}\}$ converges to zero in one norm, in the other it blows up. If ${\mathcal{X}}_{G}$ were a Banach space under $\|\cdot\|_{\mathcal{X}}$, it would be another Cauchy sequence, a contradiction.

10 Pointwise-integrals vs. Bochner integrals

Evaluation of Bochner integrals

A natural conjecture is that the Bochner integral, evaluated pointwise, is the pointwise integral; that is, if $h\in{\mathcal{L}}^{1}(Y,\mu,{\mathcal{X}})$, where ${\mathcal{X}}$ is any Banach space of functions defined on a measure space $\Omega$, then

[TABLE]

for all $x\in\Omega$. Usually, however, one is dealing with equivalence classes of functions and thus can expect the equation (21) to hold only for almost every $x$ in $\Omega$. Furthermore, to specify $h(y)(x)$, it is necessary to take a particular function representing $h(y)\in{\mathcal{X}}$

The Main Theorem implies that (21) holds for $\rho$-a.e. $x\in\Omega$ when ${\mathcal{X}}=L^{q}(\Omega,\rho)$, for $1\leq q<\infty$, is separable provided that $h=w\Phi$, where $w:Y\to{\mathbb{R}}$ is a weight function with finite $L^{1}$-norm and $\Phi:Y\to{\mathcal{X}}$ is essentially bounded, where for each $y\in Y$ $\Phi(y)(x)=\phi(x,y)$ for $\rho$-a.e. $x\in\Omega$ and $\phi:\Omega\times Y\to{\mathbb{R}}$ is $\rho\times\mu$-measurable. More generally, we can show the following.

Theorem 10.1

*Let $(\Omega,\rho)$, $(Y,\mu)$ be $\sigma$-finite measure spaces, let ${\mathcal{X}}=L^{q}(\Omega,\rho)$, $q\in[1,\infty]$, and let $h\in{\mathcal{L}}^{1}(Y,\mu;{\mathcal{X}})$ so that for each $y$ in $Y$, $h(y)(x)=H(x,y)$ for $\rho$-a.e. $x$, where $H$ is a $\rho\times\mu$-measurable real-valued function on $\Omega\times Y$. Then

(i) $y\mapsto H(x,y)$ is integrable for $\rho$-a.e. $x\in\Omega$,

(ii) the equivalence class of $x\mapsto\int_{Y}H(x,y)\,d\mu(y)\mbox{ is in }{\mathcal{X}}$, and

(iii) for $\rho$-a.e. $x\in\Omega$*

[TABLE]

**Proof. ** We first consider the case $1\leq q<\infty$. Let $g$ be in ${\mathcal{L}}^{p}(\Omega,\rho)$, where $1/p+1/q=1$. Then

[TABLE]

Here we have used Young’s inequality and Bochner’s theorem. By Fubini’s theorem, (i) follows. In addition, the map $g\mapsto\int_{\Omega}g(x)\left(\int_{Y}H(x,y)\,d\mu(y)\right)d\rho(x)$ is a continuous linear functional $F$ on $L^{p}$ with $\|F\|_{{\mathcal{X}}^{*}}\leq\int_{Y}\|h(y)\|_{\mathcal{X}}\,d\mu(y)$. Since $(L^{p})^{*}$ is $L^{q}$ for $1<q<\infty$, then the function $x\mapsto\int_{Y}H(x,y)\,d\mu(y)$ is in ${\mathcal{X}}=L^{q}$ and has norm $\leq\int_{Y}\|h(y)\|_{\mathcal{X}}\,d\mu(y)$. The case $q=1$ is covered by taking $g\equiv 1$, a member of $L^{\infty}$, and noting that $\|\int_{Y}H(x,y)\,d\mu(y)\|_{L^{1}}=\int_{\Omega}|\int_{Y}H(x,y)\,d\mu(y)|d\rho(x)\leq\int_{\Omega}\int_{Y}|H(x,y)|\,d\mu(y)d\rho(x)=\int_{Y}\|h(y)\|_{\mathcal{X}}\,d\mu(y)$. Thus, (ii) holds for $1\leq q<\infty$.

Also by Fubini’s theorem and Theorem 3.3, for all $g\in{\mathcal{X}}^{*}$,

[TABLE]

[TABLE]

Hence (iii) holds for all $q<\infty$, including $q=1$.

Now consider the case $q=\infty$. For $g\in L^{1}(\Omega,\rho)=\left(L^{\infty}(\Omega,\rho)\right)^{*}$, the inequality (23) holds, and by [18, pp. 348–9], (i) and (ii) hold and $\|\int_{Y}H(x,y)\,d\mu(y)\|_{\infty}\leq\int_{Y}\|h(y)\|_{\infty}\,d\mu(y)<\infty$. For $g\in{\mathcal{L}}^{1}(\Omega,\rho)$,

[TABLE]

[TABLE]

The two functions integrated against $g$ are in $L^{\infty}(\Omega,\rho)$ and agree, so the functions must be the same $\rho$-a.e. $\Box$

There are cases where ${\mathcal{X}}$ consists of pointwise-defined functions and (21) can be taken literally.

If ${\mathcal{X}}$ is a separable Banach space of pointwise-defined functions from $\Omega$ to ${\mathbb{R}}$ in which the evaluation functionals are bounded (and so in particular if ${\mathcal{X}}$ is a reproducing kernel Hilbert space [4]), then (21) holds for all $x$ (not just $\rho$-a.e.). Indeed, for each $x\in\Omega$, the evaluation functional $E_{x}:f\mapsto f(x)$ is bounded and linear, so by Theorem 3.3, $E_{x}$ commutes with the Bochner integral operator. As non-separable reproducing kernel Hilbert spaces exist [3, p.26], one still needs the hypothesis of separability.

In a special case involving Bochner integrals with values in Marcinkiewicz spaces, Nelson [38] showed that (21) holds. His result involves going from equivalence classes to functions, and uses a “measurable selection.” Reproducing kernel Hilbert spaces were studied by Le Page in [32] who showed that (21) holds when $\mu$ is a probability measure on $Y$ under a Gaussian distribution assumption on variables in the dual space. Another special case of (21) is derived in Hille and Phillips [19, Theorem 3.3.4, p. 66], where the parameter space is an interval of the real line and the Banach space is a space of bounded linear transformations (i.e., the Bochner integrals are operator-valued).

Essential boundedness is needed for the Main Theorem

The following is an example of a function $h:Y\to{\mathcal{X}}$ which is not Bochner integrable. Let $Y=(0,1)=\Omega$ with $\rho=\mu=\;$ Lebesgue measure and $q=1=d$ so ${\mathcal{X}}=L^{1}((0,1))$. Put $h(y)(x)=y^{-x}$. Then for all $y\in(0,1)$

[TABLE]

By l’Hospital’s rule

[TABLE]

Thus, the function $y\mapsto\|h(y)\|_{X}$ is not essentially bounded on $(0,1)$ and Theorem 12 does not apply. Furthermore, for $y\leq 1/2$,

[TABLE]

and

[TABLE]

Hence, by Theorem 3.1, $h$ is not Bochner integrable. Note however that

[TABLE]

for every $x\in\Omega$. Thus $h(y)(x)$ has a pointwise integral $f(x)$ for all $x\in(0,1)$, but $f$ is not in ${\mathcal{X}}=L^{1}((0,1))$.

Connection with sup norm

In [22], we take ${\mathcal{X}}$ to be the space of bounded measurable functions on ${\mathbb{R}}^{d}$, $Y$ equal to the product $S^{d-1}\times{\mathbb{R}}$ with measure $\nu$ which is the (completion of the) product measure determined by the standard (unnormalized) measure $d(e)$ on the sphere and ordinary Lebesgue measure on ${\mathbb{R}}$. We take $\phi(x,y):=\phi(x,e,b)=\vartheta(e\cdot x+b)$, so $x\mapsto\vartheta(e\cdot x+b)$ is the characteristic function of the closed half-space $\{x:e\cdot x+b\geq 0\}$.

We showed that if a function $f$ on ${\mathbb{R}}^{d}$ decays, along with its partials of order $\leq d$, at a sufficient rate, then there is an integral formula expressing $f(x)$ as an integral combination of the characteristic functions of closed half-spaces weighted by iterated Laplacians integrated over half-spaces. The characteristic functions all have sup-norm of 1 and the weight-function is in $L^{1}$ of $(Y,\nu)$, where $Y=S^{d-1}\times{\mathbb{R}}$ and $\nu$ is the (completion of the) product measure determined by the standard (unnormalized) measure $d(e)$ on the sphere $S^{d-1}$ of unit vectors in ${\mathbb{R}}^{d}$ and ordinary Lebesgue measure on ${\mathbb{R}}$.

For example, when $d$ is odd,

[TABLE]

where

[TABLE]

with $a_{d}$ a scalar exponentially decreasing with $d$. The integral is of the iterated directional derivative over the hyperplane with normal vector $e$ and offset $b$,

[TABLE]

For ${\mathcal{X}}={\mathcal{M}}({\mathbb{R}}^{d})$, the space of bounded Lebesgue-measurable functions on ${\mathbb{R}}^{d}$, which is a Banach space w.r.t. sup-norm, and $G$ the family $H_{d}$ consisting of the set of all characteristic functions for closed half-spaces in ${\mathbb{R}}^{d}$, it follows from Theorem 12 that $f\in{\mathcal{X}}_{G}$.

Hence, from the Main Theorem,

[TABLE]

is a Bochner integral, where $\Theta(e,b)\in{\mathcal{M}}({\mathbb{R}}^{d})$ is given by

[TABLE]

Application of the Main Theorem requires only that $w_{f}$ be in $L^{1}$, but [22] gives explicit formulas for $w_{f}$ (in both even and odd dimensions) provided that $f$ satisfies the decay conditions described above and in our paper; see also the other chapter in this book referenced earlier.

11 Some concluding remarks

Neural networks express a function $f$ in terms of a combination of members of a given family $G$ of functions. It is reasonable to expect that a function $f$ can be so represented if $f$ is in ${\mathcal{X}}_{G}$. The choice of $G$ thus dictates the $f$’s that can be represented (if we leave aside what combinations are permissible). Here we have focused on the case $G=\{\Phi(y):y\in Y\}$. The form $\Phi(y)$ is usually associated with a specific family such as Gaussians or Heavisides. The tensor-product interpretation suggests the possibility of using multiple families $\{\Phi_{j}:j\in J\}$ or multiple $G$’s to represent a larger class of $f$’s. Alternatively, one may replace $Y$ by $Y\times J$ with a suitable extension of the measure.

The Bochner integral approach also permits ${\mathcal{X}}$ to be an arbitrary Banach space (not necessarily an $L^{p}$-space). For example, if ${\mathcal{X}}$ is a space of bounded linear transformations and $\Phi(Y)$ is a family of such transformations, we can approximate other members $f$ of this Banach space ${\mathcal{X}}$ in a neural-network-like manner. Even more abstractly, we can approximate an evolving function $f_{t}$, where $t$ is time, using weights that evolve over time and/or a family $\Phi_{t}(y)$ whose members evolve in a prescribed fashion. Such an approach would require some axiomatics about permissible evolutions of $f_{t}$, perhaps similar to methods used in time-series analysis and stochastic calculus. See, e.g., [8].

Many of the restrictions we have imposed in earlier sections are not truly essential. For example, the separability constraints can be weakened. Moreover, $\sigma$-finiteness of $Y$ need not be required since an integrable function $w$ on $Y$ must vanish outside a $\sigma$-finite subset. More drastically, the integrable function $w$ can be replaced by a distribution or a measure. Indeed, we believe that both finite combinations and integrals can be subsumed in generalized combinations derived from Choquet’s theorem. The abstract transformations of the concept of neural network discussed here provide an “enrichment” that may have practical consequences.

12 Appendix I: Some Banach space background

The following is a brief account of the machinery of functional analysis used in this chapter. See, e.g., [43]. For $G\subseteq{\mathcal{X}}$, with ${\mathcal{X}}$ any linear space, let

[TABLE]

denote the set of all $n$-fold linear combinations from $G$. If the $w_{i}$ are non-negative with sum $1$, then the combination is called a convex combination; ${\rm conv}_{n}(G)$ denotes the set of all $n$-fold convex combinations from $G$. Let

[TABLE]

A norm on a linear space ${\mathcal{X}}$ is a function which associates to each element $f$ of ${\mathcal{X}}$ a real number $\|f\|\geq 0$ such that

(1) $\|f\|=0$ $\iff$ $f=0$;

(2) $\|rf\|=|r|\|f\|$ for all $r\in{\mathbb{R}};\;\mbox{and}$

(3) the triangle inequality holds: $\|f+g\|\leq\|f\|+\|g\|,\;\forall f,g\in{\mathcal{X}}$.

A metric $d(x,y):=\|x-y\|$ is defined by the norm, and both addition and scalar multiplication become continuous functions with respect to the topology induced by the norm-metric. A metric space is complete if every sequence in the space that satisfies the Cauchy criterion is convergent. In particular, if a normed linear space is complete in the metric induced by its norm, then it is called a Banach space.

Let $(Y,\mu)$ be a measure space; it is called $\sigma$-finite provided that there exists a countable family $Y_{1},Y_{2},\ldots$ of subsets of $Y$ pairwise-disjoint and measurable with finite $\mu$-measure such that $Y=\bigcup_{i}Y_{i}$. The condition of $\sigma$-finiteness is required for Fubini’s theorem. A set $N$ is called a $\mu$-null set if it is measurable with $\mu(N)=0$. A function from a measure space to another measure space is called measurable if the pre-image of each measurable subset is measurable. When the range space is merely a topological space, then functions are measurable if the pre-image of each open set is measurable.

Let $(\Omega,\rho)$ be a measure space. If $q\in[1,\infty)$, we write $L^{q}(\Omega,\rho)$ for the Banach space consisting of all equivalence classes of the set ${\mathcal{L}}^{q}(\Omega,\rho)$ of all $\rho$-measurable functions from $\Omega$ to ${\mathbb{R}}$ with absolutely integrable $q$-th powers, where $f$ and $g$ are equivalent if they agree $\rho$-almost everywhere ($\rho$-a.e.) - that is, if the set of points where $f$ and $g$ disagree has $\rho$-measure zero, and $\|f\|_{L^{q}(\Omega,\rho)}:=(\int_{\Omega}|f(x)|^{q}d\rho(x))^{1/q}$, or $\|f\|_{q}$ for short.

13 Appendix II: Some key theorems

We include, for the reader’s convenience, the statements of some crucial theorems cited in the text.

The following consequence of the Hahn-Banach Theorem, due to Mazur, is given by Yosida [43, Theorem 3’, p. 109]. The hypotheses on ${\mathcal{X}}$ are satisfied by any Banach space, but the theorem holds much more generally. See [43] for examples where ${\mathcal{X}}$ is not a Banach space.

Theorem 13.1

Let $X$ be a real locally convex linear topological space, $M$ a closed convex subset, and $x_{0}\in X\setminus M$. Then $\exists$ continuous linear functional

[TABLE]

Fubini’s Theorem relates iterated integrals to product integrals. Let $Y,Z$ be sets and ${\mathcal{M}}$ be a $\sigma$-algebra of subsets of $Y$ and ${\mathcal{N}}$ a $\sigma$-algebra of subsets of $Z$. If $M\in{\mathcal{M}}$ and $N\in{\mathcal{N}}$, then $M\times N\subseteq Y\times Z$ is called a measurable rectangle. We denote the smallest $\sigma$-algebra on $Y\times Z$ which contains all the measurable rectangles by ${\mathcal{M}}\times{\mathcal{N}}$. Now let $(Y,{\mathcal{M}},\mu)$ and $(Z,{\mathcal{N}},\nu)$ be $\sigma$-finite measure spaces, and for $E\in{\mathcal{M}}\times{\mathcal{N}}$, define

[TABLE]

where $E_{y}:=\{z\in Z:(y,z)\in E\}$ and $E^{z}:=\{y\in Y:(y,z)\in E\}$. Also, $\mu\times\nu$ is a $\sigma$-finite measure on $Y\times Z$ with ${\mathcal{M}}\times{\mathcal{N}}$ as the family of measurable sets. For the following, see Hewitt and Stromberg [18, p. 386].

Theorem 13.2

*Let $(Y,{\mathcal{M}},\mu)$ and $(Z,{\mathcal{N}},\nu)$ be $\sigma$-finite measure spaces. Let $f$ be a complex-valued ${\mathcal{M}}\times{\mathcal{N}}$-measurable function on $Y\times Z$, and suppose that at least one of the following three absolute integrals is finite: $\int_{Y\times Z}|f(y,z)|d(\mu\times\nu)(y,z)$, $\int_{Z}\int_{Y}|f(y,z)|d\mu(y)d\nu(z)$, $\int_{Y}\int_{Z}|f(y,z)|d\nu(z)d\mu(y)$. Then the following statements hold:

(i) $y\mapsto f(y,z)$ is in ${\mathcal{L}}^{1}(Y,{\mathcal{M}},\mu)$ for $\nu$-a.e. $z\in Z$;

(ii) $z\mapsto f(y,z)$ is in ${\mathcal{L}}^{1}(Z,{\mathcal{N}},\nu)$ for $\mu$-a.e. $y\in Y$;

(iii) $z\mapsto\int_{Y}f(y,z)d\mu(y)$ is in ${\mathcal{L}}^{1}(Z,{\mathcal{N}},\nu)$;

(iv) $y\mapsto\int_{Z}f(y,z)d\nu(z)$ is in ${\mathcal{L}}^{1}(Y,{\mathcal{M}},\mu)$;

(v) all three of the following integrals are equal:*

[TABLE]

[TABLE]

[TABLE]

A function $G:I\to{\mathbb{R}}$, $I$ any subinterval of ${\mathbb{R}}$, is called convex if

[TABLE]

The following formulation is from Hewitt and Stromberg [18, p. 202].

Theorem 13.3** (Jensen’s inequality)**

Let $(Y,\sigma)$ be a probability measure space. Let $G$ be a convex function from an interval $I$ into ${\mathbb{R}}$ and let $f$ be in ${\mathcal{L}}^{1}(Y,\sigma)$ with $f(Y)\subseteq I$ such that $G\circ f$ is also in ${\mathcal{L}}^{1}(Y,\sigma)$. Then $\int_{Y}f(y)d\sigma(y)$ is in $I$ and

[TABLE]

Acknowledgements

We thank Victor Bogdan for helpful comments on earlier versions.