Functional continuum regression

TL;DR

This paper introduces functional continuum regression, a new method that improves prediction accuracy in functional data analysis by encompassing and extending existing techniques like PCR and PLS.

Contribution

The paper develops a novel functional continuum regression framework that unifies and generalizes PCR and PLS, with proven consistency and improved prediction performance.

Findings

Functional CR outperforms PCR and PLS in simulations.

Functional CR provides consistent estimators.

Numerical case studies demonstrate improved accuracy.

Abstract

Functional principal component regression (PCR) can fail to provide good prediction if the response is highly correlated with some excluded functional principal component(s). This situation is common since the construction of functional principal components never involves the response. Aiming at this shortcoming, we develop functional continuum regression (CR). The framework of functional CR includes, as special cases, both functional PCR and functional partial least squares (PLS). Functional CR is expected to own a better accuracy than functional PCR and functional PLS both in estimation and prediction; evidence for this is provided through simulation and numerical case studies. Also, we demonstrate the consistency of estimators given by functional CR.

Tables

Table 1. Table 1: Time consumed (seconds) by the analysis in Section 5 with different approaches after 200 repeats (running on a laptop with Intel ® Core™ i5-5200U CPU @ 2 × \times 2.20 GHz and 8 GB RAM)

	Simulation						Medfly	Tecator™
	Scenario (i)			Scenario (ii)				Fat	Moisture	Protein
SNR	20	10	2	20	10	2
Number of curves	35	35	35	35	35	35	50	240	240	240
functional CR	94.7	91.8	95.1	68.6	69.4	68.6	176.5	2303.4	2393.9	2104.8
supervised FPCA	91.0	94.2	93.8	68.6	68.9	69.7	55.8	546.8	576.8	557.8
pFPLS	653.2	668.5	679.8	484.5	492.3	484.9	394.0	2090.3	2283.6	2029.5
FPLSR-REML	66.4	65.0	64.7	50.6	49.6	49.9	34.7	125.7	110.3	108.2
FPCRR-REML	66.0	63.7	61.9	51.3	50.9	55.0	33.6	120.4	110.3	111.3
smoothed functional PCA	56.7	59.3	60.6	47.4	47.2	48.1	34.1	241.7	207.9	207.3

Equations

$$Y=EY+{\int }_{\mathcal{T}}\beta (X-EX)+\epsilon ,$$

$$\eta (x)=E(Y|X=X_0)=EY+{\int }_{\mathcal{T}}\beta (x-EX)$$

$${\beta }_p=\underset{\theta \in \text{span}\{w_1,\dots ,w_p\}}{\mathrm{argmin}}E{\left(Y-EY-{\int }_{\mathcal{T}}\theta (X-EX)\right)}^2=\underset{\theta \in \text{span}\{w_1,\dots ,w_p\}}{\mathrm{argmin}}E{\left({\int }_{\mathcal{T}}(\beta -\theta )(X-EX)\right)}^2$$

$${\eta }_p(x)=EY+{\int }_{\mathcal{T}}{\beta }_p(x-EX),$$

$${\hat{\beta }}_p=\underset{\theta \in \text{span}\{{\hat{w}}_1,\dots ,{\hat{w}}_p\}}{\mathrm{argmin}}\frac{1}{n}\sum _{i=1}^n{\left(Y_i-\bar{Y}-{\int }_{\mathcal{T}}\theta \left(X_i-\bar{X}\right)\right)}^2$$

$${\hat{\eta }}_p(x)=\bar{Y}+{\int }_{\mathcal{T}}{\hat{\beta }}_p\left(x-\bar{X}\right),$$

$$V_X(f)(\cdot )={\int }_{\mathcal{T}}f(s)v_X(s,\cdot )ds,\forall f\in L^2(\mathcal{T}),$$

$$w_{p,\text{FPC}}=\underset{w}{\mathrm{argmax}}{\int }_{\mathcal{T}}w{V}_X(w)$$

$$\Vert w\Vert =1\text{and}{\int }_{\mathcal{T}}w{w}_{j,\text{FPC}}=0,1\le j<p.$$

$${\hat{v}}_X(s,t)=\widehat{\mathrm{cov}}(X(s),X(t))=\frac{1}{n}\sum _{i=1}^n\left(X_i(s)-\bar{X}(s)\right)\left(X_i(t)-\bar{X}(t)\right)$$

$${\hat{V}}_X(f)(\cdot )={\int }_{\mathcal{T}}f(s){\hat{v}}_X(s,\cdot )ds,\forall f\in L^2(\mathcal{T}).$$

$$w_{p,\text{FPLS}}=\underset{w}{\mathrm{argmax}}{\mathrm{cov}}^2\left(Y-{\eta }_{p-1,\text{FPLS}}(X),{\int }_{\mathcal{T}}Xw\right)$$

$$\Vert w\Vert =1\text{and}{\int }_{\mathcal{T}}w{V}_X(w_{j,\text{FPLS}})=0,1\le j<p.$$

$${\hat{w}}_{p,\text{FPLS}}=\underset{w}{\mathrm{argmax}}{\left(\frac{1}{n}\sum _{i=1}^n\left(Y_i-{\hat{\eta }}_{p-1,\text{FPLS}}(X_i)\right)\left({\int }_{\mathcal{T}}\left(X_i-\bar{X}\right)w\right)\right)}^2$$

$$\Vert w\Vert =1\text{and}{\int }_{\mathcal{T}}w{\hat{V}}_X({\hat{w}}_{j,\text{FPLS}})=0,1\le j<p,$$

$${\mathbf{w}}_{j,\alpha }=\underset{\begin{array}{c}\mathbf{w}\in {\mathbb{R}}^{d\times 1},{\mathbf{w}}^T\mathbf{w}=1\\ {\mathbf{w}}_{j,\alpha }^T{\mathbf{X}}^T\mathbf{X}\mathbf{w}=0,\forall j^{\prime }<j\end{array}}{\mathrm{argmax}}{\left({\mathbf{w}}^T{\mathbf{X}}^T\mathbf{y}\right)}^2{\left({\mathbf{w}}^T{\mathbf{X}}^T\mathbf{X}\mathbf{w}\right)}^{\frac{\alpha }{1-\alpha }-1},$$

$$w_{p,\alpha }=\underset{w}{\mathrm{argmax}}{T}_{\alpha }(w)$$

$$\Vert w\Vert =1\text{and}{\int }_{\mathcal{T}}w{V}_X(w_{j,\alpha })=0,1\le j<p,$$

$$T_{\alpha }=T_{\alpha }(w)={\mathrm{cov}}^2\left(Y,{\int }_{\mathcal{T}}Xw\right)\cdot {\left({\int }_{\mathcal{T}}w{V}_X(w)\right)}^{\frac{\alpha }{1-\alpha }-1}.$$

$${\beta }_{p,\alpha }$$

$$=\sum _{j=1}^p\left({\int }_{\mathcal{T}}\beta V_X(w_{j,\alpha })\right){\left({\int }_{\mathcal{T}}{w}_{j,\alpha }{V}_X(w_{j,\alpha })\right)}^{-1/2}{w}_{j,\alpha }$$

$${\eta }_{p,\alpha }(x)$$

$$=EY+\sum _{j=1}^p\left({\int }_{\mathcal{T}}\beta V_X(w_{j,\alpha })\right)\left({\int }_{\mathcal{T}}(x-EX)w_{j,\alpha }\right){\left({\int }_{\mathcal{T}}{w}_{j,\alpha }{V}_X(w_{j,\alpha })\right)}^{-1/2}$$

$$\underset{w}{\mathrm{maximize}}$$

$$\Vert w\Vert =1\text{and}{\int }_{\mathcal{T}}w{\hat{V}}_X({\hat{w}}_{j,\alpha })=01\le j<p,$$

$${\hat{\beta }}_{p,\alpha }$$

$$=\sum _{j=1}^p\left({\int }_{\mathcal{T}}\beta {\hat{V}}_X({\hat{w}}_{j,\alpha })\right){\left({\int }_{\mathcal{T}}{\hat{w}}_{j,\alpha }{\hat{V}}_X({\hat{w}}_{j,\alpha })\right)}^{-1/2}{\hat{w}}_{j,\alpha }$$

$$=\sum _{j=1}^p\widehat{\mathrm{cov}}\left(Y,{\int }_{\mathcal{T}}X{\hat{w}}_{j,\alpha }\right){\widehat{\mathrm{var}}}^{-\frac{1}{2}}\left({\int }_{\mathcal{T}}X{\hat{w}}_{j,\alpha }\right){\hat{w}}_{j,\alpha }$$

$${\hat{\eta }}_{p,\alpha }(X)=\bar{Y}+{\int }_{\mathcal{T}}{\hat{\beta }}_{p,\alpha }\left(x-\bar{X}\right).$$

$$w_{1,0}=\underset{w:\Vert w\Vert =1}{\mathrm{argmax}}\frac{{\mathrm{cov}}^2\left(Y,{\int }_{\mathcal{T}}Xw\right)}{{\int }_{\mathcal{T}}w{V}_X(w)}=\underset{w:\Vert w\Vert =1}{\mathrm{argmax}}\frac{{\mathrm{cov}}^2\left(Y,{\int }_{\mathcal{T}}Xw\right)}{\mathrm{var}(Y)\mathrm{var}\left({\int }_{\mathcal{T}}Xw\right)}.$$

$$\mathrm{cov}\left(Y-{\eta }_{p-1,\frac{1}{2}}(X),{\int }_{\mathcal{T}}Xw\right)=\mathrm{cov}\left(Y,{\int }_{\mathcal{T}}Xw\right).$$

$$X^{(p,\alpha )}=X-EX-\sum _{j=1}^{p-1}\left({\int }_{\mathcal{T}}(X-EX)w_{j,\alpha }\right){\left({\int }_{\mathcal{T}}{w}_{j,\alpha }{V}_X(w_{j,\alpha })\right)}^{-1/2}{V}_X(w_{j,\alpha })$$

$$Y^{(p,\alpha )}=Y-{\eta }_{p-1,\alpha }(X)={\int }_{\mathcal{T}}\beta X^{(p,\alpha )}.$$

Full text

††journal: Journal of Multivariate Analysis

url]http://www.sfu.ca/ zza115/

1 Introduction

1.1 Scalar-on-function regression model

With the development of technology, the demand for functional data analysis (FDA) is increasing; it is frequent to encounter data that are recorded continuously within a nondegenerate and compact interval ${\mathcal{T}}$. Scalar-on-function regression models link the scalar response $Y$ to the integral of product of random process $X=X(t)$, $t\in{\mathcal{T}}$, and corresponding unknown fixed function $\beta=\beta(t)$. To be precise, the linkage is of the form that

[TABLE]

where $\int_{\mathcal{T}}f$ is short for the Lebesgue integral $\int_{\mathcal{T}}f(t)\operatorname*{d\!}t$ and where $\operatorname*{E}(\varepsilon|X)=0$, $\operatorname*{cov}(\varepsilon,X)=0$, and $\int_{\mathcal{T}}\operatorname*{E}X^{2}=\operatorname*{E}\left(\int_{\mathcal{T}}X^{2}\right)<\infty$. All the functions involved in this paper are assumed to be square-integrable, i.e., the discussion is limited to $L^{2}(\mathcal{T})$ (or $L^{2}(\mathcal{T}\times\mathcal{T})$), the $L^{2}$-space on $\mathcal{T}$ (or $\mathcal{T}\times\mathcal{T}$) with respect to the Lebesgue measure. In addition, $\|\cdot\|$ stands for the $L^{2}$-norm, i.e., $\|f\|$ equals $\sqrt{\int_{\mathcal{T}}f^{2}}$ for $f\in L^{2}(\mathcal{T})$ and $\sqrt{\int_{\mathcal{T}}\int_{\mathcal{T}}f^{2}}$ if $f\in L^{2}(\mathcal{T}\times\mathcal{T})$.

The infinite-dimensional structure of functional space makes data analysis challenging: the dimension of parameter space exceeds the number of observed subjects, and hence dimension-reduction techniques are indispensable in model-fitting. To estimate the coefficient function $\beta$ and the conditional expectation

[TABLE]

for $x$, a realization of $X$, the standard approach is to express $\beta$ in terms of a finite set of functions $\{w_{1},\ldots,w_{p}\}$ truncated from basis functions $\{w_{1},w_{2},\ldots\}\subset L^{2}(\mathcal{T})$. This inspires us to approximate $\beta$ and $\eta(x)$, respectively, by

[TABLE]

and

[TABLE]

where $\text{span}\{w_{1},\ldots,w_{p}\}$ is the linear space spanned by $w_{1},\ldots,w_{p}$. Note that $\beta_{p}$ is the slope of the best approximation (restricted within $\text{span}\{w_{1},\ldots,w_{p}\}$ and in the $L^{2}$-sense) to $Y$ by a linear function of $X$. In particular, $\beta_{p}=\sum_{j=1}^{p}\left(\int_{\mathcal{T}}\beta w_{j}\right)w_{j}$ if $\{w_{1},\ldots,w_{p}\}$ is orthonormal (i.e., $\int_{\mathcal{T}}w_{j}^{2}=1$ for $1\leq j\leq p$ and $\int_{\mathcal{T}}w_{j}w_{j^{\prime}}=0$ for $j\neq j^{\prime}$). In addition, for the completeness of definition, write $\beta_{0}=0$ and $\eta_{0}(x)=\operatorname*{E}Y$.

Though it is possible to employ a basis independent of the data (e.g., polynomial basis, Fourier basis, etc.), it is more reasonable to force the basis adapt to data. In that case, $w_{1},,w_{2},\ldots$ are often unknown apriori and need to be replaced with corresponding estimates $\hat{w}_{1},\hat{w}_{2},\ldots$. Let $(X_{i},Y_{i})$, $i=1,\ldots,n$, be $n$ pairs of independently observed data, all distributed as $(X,Y)$. Correspondingly, we have estimates for (1) and (2), respectively,

[TABLE]

and

[TABLE]

where $\bar{X}=n^{-1}\sum_{i=1}^{n}X_{i}$ and $\bar{Y}=n^{-1}\sum_{i=1}^{n}Y_{i}$.

Under this framework, the accuracy of estimates $\hat{\beta}_{p}$ and $\hat{\eta}_{p}(x)$ varies with the choice of $\{\hat{w}_{1},\hat{w}_{2},\ldots\}$. Two well-known options are discussed in Section 1.2, leading to functional principal component regression (PCR) and functional partial least squares (PLS), respectively.

1.2 Functional principal component and functional partial least squares bases

Among all the dimension-reduction techniques, functional PCR is the most prevailing one. It is built on the functional principal component basis, say $\{w_{1,\text{FPC}},w_{2,\text{FPC}},\ldots\}$, constructed from covariance operator $V_{X}:L^{2}(\mathcal{T})\rightarrow L^{2}(\mathcal{T})$ given by

[TABLE]

where $v_{X}(s,t)=\operatorname*{cov}(X(s),X(t))$. The assumption $v_{X}\in L^{2}(\mathcal{T}\times\mathcal{T})$ implies that $V_{X}$ is of Hilbert-Schmidt class and hence possesses a countable number of eigenvalues, all real and nonnegative. Specifically, $w_{p,\text{FPC}}$ is taken as the $p$-th eigenfunction of $V_{X}$, or equivalently, given $w_{1,\text{FPC}},\ldots,w_{p-1,\text{FPC}}$,

[TABLE]

subject to

[TABLE]

Empirically, substitute

[TABLE]

for $v_{X}(s,t)$ and then estimate $w_{p,\text{FPC}}$ by $\hat{w}_{p,\text{FPC}}$, the $p$-th eigenfunction of operator $\widehat{V}_{X}$ satisfying that

[TABLE]

During the past few decades, extensive work has focused on functional PCR; more details can be found in a number of monographs (e.g., Ramsay and Silverman [24] and Horváth and Kokoszka [15]) and review papers (e.g., Wang et al. [32] and Febrero-Bande et al. [12]).

As defined in (4), the construction of the functional principal component basis is “unsupervised” due to no involvement of $Y$; the first $p_{0}$ elements of this basis seek to explain most of the variation of $X$, whereas they are not necessarily important in representing $\beta$. That is, it is possible for one or more elements in the abandoned part $\{w_{p+1,\text{FPC}},\ldots\}$ to be highly correlated with the response.

Some efforts have already been made to target this well-known defect, including Preda and Saporta [22] who extended the multivariate PLS to functional PLS. This technique relies on functional PLS basis which is defined in a sequential manner: given $w_{1,\text{FPLS}},\ldots,w_{p-1,\text{FPLS}}$,

[TABLE]

subject to

[TABLE]

The corresponding empirical version is

[TABLE]

subject to

[TABLE]

where functions $\eta_{{p-1},\text{FPLS}}$ and $\hat{\eta}_{{p-1},\text{FPLS}}$ are respective counterparts of (2) and (3).

Functional PLS has been later investigated and developed by, for instance, Reiss and Ogden [26], Delaigle and Hall [11], Aguilera et al. [1]. PLS and its derivatives are referred to as “fully supervised” and may suffer the “double-dipping” problem: they employ the covariance between $Y$ and $X$ both for the construction of basis functions and for further prediction; the resulting findings are possibly vulnerable and sensitive to small signals; see [17]. Nie et al. [21] attempted to put forward a linear combination of functional PCR and functional PLS; their proposal lies between unsupervised and fully supervised techniques. Different from these authors, we borrow the idea of continuum regression (CR) from Stone and Brooks [30] and extend it to the learning of functional data.

1.3 Continuum regression

Briefly, in the context of multivariate analysis with response $\bm{y}\in\mathbb{R}^{n\times 1}$ and design matrix $\mathbf{X}\in\mathbb{R}^{n\times d}$ both column-mean-centered, CR projects $\bm{y}$ to the linear space spanned by mutually orthogonal regressors $\mathbf{X}\bm{w}_{1,\alpha},\ldots,\mathbf{X}\bm{w}_{p,\alpha}\in\mathbb{R}^{n\times 1}$, after successively computing

[TABLE]

where $\alpha\in[0,1)$ and $p$ ($\leq d$) are to be tuned. The most appealing property of CR, as proved by Stone and Brooks [30], is that its framework encompasses ordinary least square (OLS) ($\alpha=0$), PLS ($\alpha=1/2$), and PCR ($\alpha\rightarrow 1$). Accordingly, the model resulting from CR is expected to outperform those from OLS, PLS, and PCR in terms of prediction.

There have been some further developments of CR. Sundberg [31] connected it to the ridge regression. Björkström and Sundberg [3] and Jung [17] revealed the analytical form of (6). Lee and Liu [19] combined CR with the kernel learning to accommodate the nonlinear regression. Chen and Cook [6] proved the possible inconsistency of estimators produced by CR, while Chen and Zhu [7] showed the consistency given by CR in estimating the central (dimensional-reduction) subspace defined by Cook [8] and Cook [9, pp. 105].

In the remainder of this paper, Section 2 introduces functional CR and its special cases. Our consistency results are presented in Section 3, based on which Section 4 derives an effective algorithm. Empirical evidence appears in Section 5, where our method is compared with existing ones in terms of both estimation and prediction. Section 6 discusses of pros and cons of functional CR as well as possible future work. For the sake of brevity, technical details are left in appendices.

2 Functional continuum regression

2.1 Functional continuum basis

We begin by defining the functional continuum basis which will be denoted by $\{w_{1,\alpha},w_{2,\alpha},\ldots\}$. For a pre-determined $\alpha\in[0,1)$, we construct the basis in a sequential way. Given $w_{1,\alpha},\ldots,w_{p-1,\alpha}$, define

[TABLE]

subject to

[TABLE]

where

[TABLE]

Following (1) and (2),

[TABLE]

and

[TABLE]

are resulting approximations to $\beta$ and $\eta(x)$, respectively.

Having defined the population version of the functional continuum basis functions, we now give the empirical counterpart. The empirical version, say $\{\hat{w}_{1,\alpha},\hat{w}_{2,\alpha},\ldots\}$, is defined recursively. Once the first $j-1$ elements are determined, $\hat{w}_{p,\alpha}$ is taken as the maximizer of following optimization problem:

[TABLE]

where operator $\widehat{V}_{X}$ is defined as (5). Further, $\beta_{p,\alpha}$ and $\eta_{p,\alpha}(x)$ are respectively estimated by

[TABLE]

and

[TABLE]

Return to the definition of $w_{p,\alpha}$ in (7). Though it looks like a natural extension of that of the $d$-vector (6), at least two concerns (Propositions 1 and 2) arise with the non-concavity of objective functions $T_{\alpha}(w)$ and $\widehat{T}_{\alpha}(w)$ and the infinite dimension of $L^{2}(\mathcal{T})$: one is the existence of $w_{p,\alpha}$ and $\hat{w}_{p,\alpha}$ which is not trivial at all since the unit sphere and unit ball in $L^{2}(\mathcal{T})$ are no longer compact; the other is whether or not, for any preset $\alpha\in[0,1)$, $\beta$ can be fully expressed in terms of the functional continuum basis $\{w_{1,\alpha},w_{2,\alpha},\ldots\}$.

Proposition 1**.**

Given $w_{1,\alpha},\ldots,w_{p-1,\alpha}$, the objective function $T_{\alpha}$ defined in (9), subject to conditions (8), has a maximizer. So does $\widehat{T}_{\alpha}$ in (10) with fixed $\hat{w}_{1,\alpha},\ldots,\hat{w}_{p-1,\alpha}$.

Proposition 2**.**

Suppose $\beta$ can be expansed in terms of eigenfunctions of $V_{X}$. Then, for each $\alpha\in[0,1)$, $\beta$ belongs to $\overline{\text{span}\{w_{1,\alpha},w_{2,\alpha},\ldots\}}$, the closure of $\text{span}\{w_{1,\alpha},w_{2,\alpha},\ldots\}$.

2.2 Special cases

Functional CR inherits the inclusion property of CR; i.e., for certain $\alpha$’s, functional CR reduces to some existing methods.

Firstly, as $\alpha\rightarrow 1$, the variance term $\int_{\mathcal{T}}wV_{X}(w)$ dominates the objective function (9) and the role of $\operatorname*{cov}(Y,\int_{\mathcal{T}}Xw)$ is negligible. We assert that, in this scenario, the functional continuum basis is identical to the functional principal component basis.

Proposition 3**.**

If $\operatorname*{cov}\left(Y,\int_{\mathcal{T}}Xw_{p,\text{FPC}}\right)\neq 0$ for all $p\in\mathbb{N}$, then $w_{p,\alpha}=w_{p,\text{FPC}}$ as $\alpha\rightarrow 1$.

At the other extreme ($\alpha=0$), note that

[TABLE]

Geometrically, $w_{1,\alpha}$ maximizes the squared cosine of angle between $\int_{\mathcal{T}}Xw$ and $Y$. Therefore, $\int_{\mathcal{T}}Xw_{1,0}$ is parallel to the orthogonal projection of $Y$ onto $X$, meaning that $\operatorname*{cov}\left(Y,\int_{\mathcal{T}}Xw\right)$ must be zero for all $w$ such that $\int_{\mathcal{T}}wV_{X}(w_{1,0})=0$. That is to say, the sequential construction terminates at $w_{1,\text{FCR}}$ and no subsequent element exists. Obviously, in this situation, functional CR is equivalent to a functional version of OLS regression.

Another special case lies midway between two extremes, i.e., $\alpha=1/2$. Noticing that under the need of constraints (8), we have

[TABLE]

One can see that this case is identical to the functional PLS introduced in Section 1.2.

3 Theoretical properties

3.1 Equivalent forms of the functional continuum basis

Considering residuals of $X$ and $Y$ after the first $p-1$ steps, we merge the last $p-1$ side-conditions in (8) and the objective function (7) together. This reformulation simplifies forthcoming proofs and facilitates the implementation in Section 4 as well.

Proposition 4**.**

Given $w_{1,\alpha},\ldots,w_{p-1,\alpha}$ with $\int_{\mathcal{T}}w_{j,\alpha}V_{X}(w_{j,\alpha})>0$ for $j=1,\ldots,p-1$, write

[TABLE]

and

[TABLE]

Then, $w_{p,\alpha}$ defined in (7) can be found by maximizing $T_{p,\alpha}^{*}$ on the unit sphere, i.e.,

[TABLE]

where

[TABLE]

An empirical counterpart of Proposition 4 naturally follows.

Proposition 5**.**

Given $\hat{w}_{1,\alpha},\ldots,\hat{w}_{p-1,\alpha}$ with $\int_{\mathcal{T}}\hat{w}_{j,\alpha}\widehat{V}_{X}(\hat{w}_{j,\alpha})>0$ for all $j\leq p-1$, write

[TABLE]

and

[TABLE]

for $i=1,\ldots,n$. Then,

[TABLE]

where

[TABLE]

with $\widehat{V}_{\widehat{X}^{(p,\alpha)}}=\widehat{V}_{\widehat{X}^{(p,\alpha)}}(s,t)=n^{-1}\sum_{i=1}^{n}\widehat{X}_{i}^{(p,\alpha)}(s)\widehat{X}_{i}^{(p,\alpha)}(t)$.

Previously, both in (7) and (14), the functional continuum basis has been defined as a set of maximizers of sequential optimization problems. Proposition 6 derives an alternative but more explicit form of these desired solutions: they are constructed by adjusting the projection of function $\beta$ on some directions.

Proposition 6**.**

Given $\alpha\in[0,1)$, $p\in\mathbb{N}$, and $w_{1,\alpha},\ldots,w_{p-1,\alpha}$. Let $\lambda_{j}^{(p,\alpha)}$ denote the $j$-th largest eigenvalue of $V_{X^{(p,\alpha)}}$ with corresponding eigenfunction $\phi_{j}^{(p,\alpha)}$. Suppose $\lambda_{1}^{(p,\alpha)}$ has multiplicity $m\geq 1$, i.e., $\lambda_{1}^{(p,\alpha)}=\cdots=\lambda_{m}^{(p,\alpha)}>\lambda_{m+1}^{(p,\alpha)}$. If $V_{X^{(p,\alpha)}}(\beta)$ is not orthogonal to $\text{span}\left\{\phi_{1}^{(p,\alpha)},\ldots,\phi_{m}^{(p,\alpha)}\right\}$, there exists $\delta^{(p,\alpha)}\in(-1,0)\cup(0,\infty)$ such that

[TABLE]

where . The three boundary values of $\delta^{(p,\alpha)}$, $\{-1,0,\infty\}$, correspond to functional PCR ($\delta^{(p,\alpha)}\rightarrow-1$), functional PLS ($\delta^{(p,\alpha)}\rightarrow 0$) and functional OLS ($\delta^{(p,\alpha)}\rightarrow\infty$), respectively.

3.2 Consistency of the empirical functional continuum basis and corresponding estimators

We need two more conditions:

1. (C1)

For each $j$ ($\leq p$), $T_{j,\alpha}^{*}(w)$ attains a unique maximizer (up to sign) in $\{w\in L^{2}(\mathcal{T}):\|w\|=1\}$. 2. (C2)

$\operatorname*{E}\|X\|^{4}<\infty$.

Our main result, Theorem 1, demonstrates the consistency of estimators in the case of “fixed $p$ and infinite $n$”.

Theorem 1**.**

Fix $\alpha$ and $p$. Under (C1), $\|\hat{w}_{p,\alpha}-w_{p,\alpha}\|\stackrel{{\scriptstyle\text{P}}}{{\longrightarrow}}0$ as $n\rightarrow\infty$. If we also have (C2) apart from (C1), then $\|\hat{\beta}_{p,\alpha}-\beta_{p,\alpha}\|$ and $|\hat{\eta}_{p,\alpha}(x)-\eta_{p,\alpha}(x)|$ both converge to zero in probability as $n$ diverges, where $x$ is a realization of $X$ and independent from $X_{1},\ldots,X_{n}$.

Remark 1**.**

We do not have to impose uniqueness on the empirical version $\operatorname*{arg\,max\,}_{\|w\|=1}\widehat{T}_{j,\alpha}^{*}(w)$ for all $j\leq p$. If $\operatorname*{arg\,max\,}_{\|w\|=1}\widehat{T}_{j,\alpha}^{*}(w)$ is not unique, the proof of Theorem 1 is still valid as long as the chosen $\hat{w}_{j,\alpha}$ is measurable. Jennrich [16, Lemma 2] provided a way to find such a measurable $\hat{w}_{j,\alpha}$.

4 Implementation

We understand that, when facing real data, it is not feasible to calculate the integrals exactly; instead, one may replace all integrals with discrete summations after expanding $X_{1},\ldots,X_{n}$ in terms of finite basis functions (e.g., B-splines). Though this approximation definitely affects the accuracy of resulting estimators, the corresponding discussion is out of the scope of this paper. Therefore, we will keep the integral notations, even in the description of implementation.

It is feasible to duplicate the ideas of Lee and Liu [19] and Chan and Mak [5] to solve maximization problem (10) through a univariate root-finding procedure. Nevertheless, the implementation would be more natural and straightforward if we apply the following identity, an empirical version of Proposition 6.

Proposition 7**.**

Fix $\hat{w}_{1,\alpha},\ldots,\hat{w}_{p-1,\alpha}$. Let $\hat{\lambda}_{j}^{(p,\alpha)}$ is the $j$-th largest eigenvalue of $\widehat{V}_{\widehat{X}^{(p,\alpha)}}$ with corresponding eigenfunction $\hat{\phi}_{j}^{(p,\alpha)}$. Suppose $\hat{\lambda}_{1}^{(p,\alpha)}=\cdots=\hat{\lambda}_{m}^{(p,\alpha)}>\hat{\lambda}_{m+1}^{(p,\alpha)}$. If $\widehat{V}_{\widehat{X}^{(p,\alpha)}}(\beta)=n^{-1}\sum_{i=1}^{n}\widehat{X}_{i}^{(p,\alpha)}\widehat{Y}_{i}^{(p,\alpha)}$ is not orthogonal to $\text{span}\left\{\hat{\phi}_{1}^{(p,\alpha)},\ldots,\hat{\phi}_{m}^{(p,\alpha)}\right\}$, there is $\hat{\delta}^{(p,\alpha)}\in(-1,0)\cup(0,\infty)$ such that

[TABLE]

Remark 2**.**

It is indispensable to assume that $\widehat{V}_{\widehat{X}^{(p,\alpha)}}(\beta)$ is not orthogonal to $\text{span}\left\{\hat{\phi}_{1}^{(p,\alpha)},\ldots,\hat{\phi}_{m}^{(p,\alpha)}\right\}$ when $\hat{\lambda}_{1}^{(p,\alpha)}=\cdots=\hat{\lambda}_{m}^{(p,\alpha)}>\hat{\lambda}_{m+1}^{(p,\alpha)}$. So Proposition 7 does not cover all the possibilities; the ridge type solution may be not a global maximizer when the assumption is violated. Exceptions are constructible artificially, yet they are rare in practice [3, 17], especially when $\varepsilon$ and $X(t)$ with given $t$ are all continuously distributed. Actually, if the assumption is not fulfilled, one can always project $\widehat{X}_{i}^{(p,\alpha)}$ onto the (orthogonal) compliment of $\text{span}\left\{\hat{\phi}_{1}^{(p,\alpha)},\ldots,\hat{\phi}_{m}^{(p,\alpha)}\right\}$ and take the projection as the substitute for $\widehat{X}_{i}^{(p,\alpha)}$.

Proposition 7 suggests merely considering $w$ in the form of (16). It helps us narrow down the search scope for $\hat{w}_{p,\alpha}$ by reformulating (7) into a univariate maximization problem. The only unknown item in (16), $\hat{\delta}^{(p,\alpha)}$, is taken as

[TABLE]

where we obtain

[TABLE]

by substituting the right-hand side of (16) for $w$ in (15). The univariate function $\ln Q_{p,\alpha}$ depends on not only $p$ and $\alpha$ but also observed data, which makes it inconvenient to theoretically investigate this function’s plot. However, for the specific datasets to be investigated in Section 5, there seems no more than one local maximum within either $(-1,0)$ or $(0,\infty)$; see Figure 1. As a result, the maximization in each piece is able to be handled by arbitrary symbolic computation program.

To reduce computational burden and increase the efficiency of Algorithm 1, we compute $\widehat{X}_{i}^{(p,\alpha)}$ and $\hat{\beta}_{p,\alpha}$ in a recursive way, namely, for $i=1,\ldots,n$,

[TABLE]

and

[TABLE]

starting with $\widehat{X}_{i}^{(1,\alpha)}=X_{i}-\bar{X}$, $\widehat{Y}_{i}^{(1,\alpha)}=Y_{i}-\bar{Y}$ and $\hat{\beta}_{0,\alpha}=0$.

4.1 Tuning parameters

The result of functional CR relies on the choice of two parameters: $\alpha$, the continuum parameter, and $p$, the number of basis functions included in the model. Favoring a much lower expense in computation, we tune them through the generalized cross-validation (GCV, Craven and Wahba [10]). For each possible pair $(p,\alpha)$, a GCV-type criterion employed here is

[TABLE]

The minimizer of $\text{GCV}(p,\alpha)$ is chosen as the optimal combination; see Algorithm 1 for details.

5 Numerical illustration

To illustrate the performance of functional CR, the result given by our method is compared with those from supervised FPCA (Nie et al. [21]), pFPLS (Aguilera et al. [1]), FPLSR-, FPCRR-REML (both recommended by Reiss and Ogden [26] after a series of comparisons) and smoothed functional PCA (Ramsay and Silverman [24, Section 9.3]). Among these the first four are supervised, while the other two are categorized as unsupervised.

With the aid of R (R Core Team [23]), RStudio™ (RStudio Team [27]) and R-package fda (Ramsay et al. [25]), we code all the methods mentioned in the preceding paragraph except for FPCRR-REML (implemented by R-function fpcr coded by its proposers and included in R-package refund jointly created by Goldsmith et al. [14]).

5.1 Simulation study

The dataset CanadianWeather in Ramsay et al. [25] contains the (base 10 logarithm of) precipitation at 35 different locations in Canada averaged over 1960 to 1994. We extract the mean function $\mu_{X}$ and the top $j$-th eigenvalue $\lambda_{j}$ and eigenfunction $w_{j,\text{FPC}}$ of the covariance operator, $j=1,2,3$, from this dataset. Each sample in this simulation consists of 35 artificial curves on $\mathcal{T}=[1,365]$ of the following form:

[TABLE]

where $\xi_{ij}$ follows $N(0,\lambda_{j})$ independently and 35 is the number of curves included in CanadianWeather. Further, $Y_{i}$ are generated as

[TABLE]

with $\varepsilon_{i}\overset{\text{iid}}{\sim}N(0,\sigma)$. The quantity $\int_{\mathcal{T}}\mu_{X}\beta/\sigma$ is informally referred to as the signal-to-noise-ratio (SNR).

We consider two sorts of coefficient function coupled with three levels of SNR (2,10 and 20):

1. (i)

$\beta=w_{1,\text{FPC}}$; 2. (ii)

$\beta=w_{3,\text{FPC}}$.

No matter how supervised they are, all the methods are applicable to Scenario (i), while Scenario (ii) is to imitate the situation where the coefficient function is orthogonal to the top few eigenfunctions of $V_{X}$, i.e., the initial target our proposal is shooting at. For each combination of $\beta$ and SNR, we generate 200 samples and apply the six techniques to each dataset to estimate $\beta$. The estimation quality is directly measured by the square root of mean squared errors (RMSE) (on each $t\in\mathcal{T}$) of estimated coefficient functions.

The candidate pool of $(p,\alpha)$ for functional CR is a $2\times 11$ grid, $\{1,2\}\times\{0,.1,.2,.3,.4,.5,.6,.7,.8,.9,.999\}$, where the scope of $p$, $\{1,2\}$, remains for all five other methods. This setting is reasonable because the first two functional principal components capture around 97% of the total variation in predictor. In the implementation of smoothed functional PCA, supervised FPCA and pFPLS, smoothing penalty parameters are chosen from $\{0,1,10,10^{2},10^{3},10^{4},10^{5}\}$. Moreover, as suggested by Nie et al. [21], candidate values of the “weight” parameter needed by supervised FPCA are taken from $\{0,.1,.2,.3,.4,.5,.6,.7,.8,.9,1\}$.

When $\beta=w_{1,\text{FPC}}$ and SNR is more than moderate ($\text{SNR}=10,20$), functional CR (solid red) performs as well as pFPLS and smoothed functional PCA and slightly better than FPLSR-REML and FPCRR-REML whose RMSEs become dramatically high at the two extremes of the domain. For any method, RMSEs are enlarged with a decrease of SNR, while functional CR (solid red) seems more sensitive to the change of SNR; see Figure 2.

Unsurprisingly, as shown in Figure 3, Scenario (ii) ($\beta=w_{3,\text{FPC}}$) does not favor the smoothed functional PCA which involves only two basis functions nearly orthogonal to $\beta$. Oppositely, functional CR (solid red) outperforms competitors regardless of SNR and returns the lowest RMSE at almost every $t\in\mathcal{T}$, in spite of the setting violating the assumption of Proposition 6. Note that curves for supervised FPCA are never included in Figures 2 and 3 as RMSEs from supervised FPCA are much larger than those from other approaches; either the estimators from Nie et al. [21] are not consistent or it may need a larger sample size to reach a more satisfying accuracy.

5.2 Application to real data

For each of following two datasets, we randomly take roughly 10% of all the samples of each dataset for testing and the remaining for training. Repeat the random split for 200 times. To alleviate impacts from different testing sets and facilitate the comparison in prediction, define the relative mean squared prediction error (ReMSPE):

[TABLE]

where $I_{\text{train}}$ and $I_{\text{test}}$ are respective index sets for training and testing data, $\#I_{\text{train}}$ is the cardinality of $I_{\text{train}}$, and $\widehat{Y}_{i}$ is the prediction corresponding to $Y_{i}$. For each approach, generate a boxplot of the 200 ReMSPEs. As for the candidate pool for tuning parameters, we keep all the settings in Section 5.1 except the one for $p$; we raise its upper bound from 2 to 5 to accommodate the new datasets.

5.2.1 Medfly data

Investigated in substantial literature (see, e.g., Müller et al. [20] and Sang et al. [28]), the Mediterranean fruit fly, or medfly for short, has become indeed a popular object of study, partly owing to its short lifespan. Posted at http://faculty.bscb.cornell.edu/~hooker/FDA2008/medfly.Rdata, the medfly data here records lifespans of 50 female flies as well as numbers of eggs laid by each of them in each of the 26 days. People would like to uncover how lifespan is influenced by fecundity as time goes on by bridging the curves of egg count to lifespans.

Taking the egg count and lifespan as predictor and response respectively, all the six methods, no matter whether supervised or unsupervised, perform fairly close to each, though FPCRR- and FPLSR-REML appear way better than the other four; see Figure 4.

5.2.2 Tecator™ data

A Tecator™ Infratec Food and Feed Analyzer recorded near infrared absorbance spectra (ranging from 850 to 1050 nm and divided into 100 channels) of 240 finely chopped pure meat samples with different fat, moisture and protein contents. The dataset is now publicly accessible at http://lib.stat.cmu.edu/datasets/tecator, containing the absorbance spectra (i.e., the logarithm to base 10 of transmittance at each wavelength) and the three contents measured in percent by analytic chemistry.

We regress the fat, moisture and protein contents, respectively, on the absorbance spectra. In the first case (spectra vs. fat), the six approaches are roughly categorized into three groups in each subfigure of Figure 5: functional CR on the left end, the three in the middle (including supervised FPCA, pFPLS and FPLSR-REML), and another two on the very right (i.e., FPCRR-REML and smoothed functional PCA). As shown in Figure 5, supervised strategies are more favored than unsupervised ones. In the cases of moisture (Figure 6) and protein (Figure 7), this phenomenon does not hold, but functional CR still takes the lead in terms of lower ReMSPEs, even though we do not impose any penalty on the smoothness of functional continuum basis functions.

6 Conclusion and discussion

Specially designed for scalar-on-function regression models, the framework of functional CR encompasses the well-known functional PCR and functional PLS, etc.. We have given various equivalent forms of functional continuum basis functions which lower the difficulty of optimization in the numerical implementation. The consistency of estimators is demonstrated for the case of fixed $p$. Verified in numerical studies and compared with five existing methods, our strategy is overall competitive in terms of both estimation and prediction.

However, our work is far from perfect. The core of our algorithm is to locate the constrained global maximizer of $\ln Q_{p,\alpha}(\delta)$. In Section 5, thanks to the simpleness of curves of $\ln Q_{p,\alpha}$, we do not have to initiate the maximization with multiple start points. Even so, our implementation is still more involved than competitors when number of curves becomes larger; see Table 1 for the time consumed for each method. But it can always be worse; it is possible for other datasets to be coupled with more complex curves for $\ln Q_{p,\alpha}$. In such cases, we have to avoid the search being trapped in some local maxima. We suggest using multiple initial values, a commonly adopted strategy. But this significantly slow down the implementation of functional CR. For instance, under the same computing environment, if we try 100 initial points in each maximization, the seconds consumed by functional CR for the Tecator™ data would be over 30 times as many as the corresponding number posted in Table 1.

Last but not least, functional CR possesses the potential to be further extended. With a generalization analogous to that in Brooks and Stone [4], it is hopeful to handle multiple responses simultaneously and even functional response. Another possible direction of evolution is to enhance the robustness by replacing variance and covariance terms with robust counterparts; just like Serneels et al. [29] did for CR.

Acknowledgment

We are grateful to the editor, associate editor and anonymous reviewers for their extremely valuable comments and constructive suggestions. Also, we would like to thank the Natural Sciences and Engineering Research Council of Canada (NSERC) for financial supports.

Appendix A Lemmas

Lemma 1 is the cornerstone of the existence of $w_{p,\alpha}$ and $\hat{w}_{p,\alpha}$. Lemma 2, essential in proving Theorem 1, reveals the convergence from the empirical objective function $\widehat{T}_{p,\alpha}^{*}$ to the theoretical one $T_{p,\alpha}^{*}$. Their proofs are both left in B.

Lemma 1**.**

Suppose $C\subseteq L^{2}(\mathcal{T})$ is a bounded and weakly sequentially closed set. Suppose $f:C\rightarrow\mathbb{R}$ is weakly sequentially upper semi-continuous. Then $f$ has a maximizer on $C$.

Remark 3**.**

Although the assumption of Lemma 1 can be further relaxed, this less general version suffices for our needs in this paper; see Theorem 5.3 and Remark 5.4 in an unpublished 2013 technical report by A. Alexanderian (available at https://aalexan3.math.ncsu.edu/articles/hilbert.pdf).

Lemma 2**.**

Recall $T_{p,\alpha}^{*}(w)$ and $\widehat{T}_{p,\alpha}^{*}(w)$ both defined in Proposition 5. In case $\hat{w}_{j,\alpha}$ converges to $w_{j,\alpha}$ in probability as $n$ goes to $\infty$ for $j=1,\ldots,p-1$, $\widehat{T}_{p,\alpha}^{*}(w)$ converges to $T_{p,\alpha}^{*}(w)$ in probability uniformly over the unit ball, i.e.,

[TABLE]

Appendix B Proofs

Proof of Lemma 1.

Firstly prove that $f_{0}=\sup_{x\in C}f(x)<\infty$. To the contrary, suppose that $f_{0}=\infty$. Then there is a sequence $\{x_{n}\}\subset C$ such that $f(x_{n})\geq n$ for each $n\in\mathbb{N}$. Deduced from the boundedness and weakly sequential closeness, the weakly sequential compactness of $C$ implies that $\{x_{n}\}$ must have a subsequence $\{x_{n_{k}}\}$ which weakly converges to $x^{*}\in C$. Due to the weakly sequential upper semi-continuity of $f$, we have

[TABLE]

This identity contradicts the range of $f$.

Next, there always exists a sequence $\{x_{n}\}$ such that $\lim_{n\rightarrow\infty}f(x_{n})=f_{0}$. Find a weakly convergent sequence $\{x_{n_{k}}\}\subseteq\{x_{n}\}$ with limit $x^{*}\in C$. Thus,

[TABLE]

The sandwich rule indicates that $x^{*}\in C$ is a maximizer of $f$ on $C$ and completes this proof. ∎

Proof of Lemma 2.

The proof consists of three steps. First follow Delaigle and Hall [11, Eq. (5.1)] to conclude that, as $n\rightarrow\infty$,

[TABLE]

Moreover, $\forall\varepsilon>0$, $\exists\delta>0$ such that

[TABLE]

and

[TABLE]

The continuous mapping theorem guarantees the convergence in probability of $\widehat{X}^{(p,\alpha)}(t)$ to $X^{(p,\alpha)}(t)$ and further yields that

[TABLE]

Recall $\widehat{V}_{\widehat{X}^{(p,\alpha)}}=\widehat{V}_{\widehat{X}^{(p,\alpha)}}(s,t)=n^{-1}\sum_{i=1}^{n}\widehat{X}_{i}^{(p,\alpha)}(s)\widehat{X}_{i}^{(p,\alpha)}(t)$ and $\widehat{V}_{\widehat{X}^{(p,\alpha)}}(\beta)=n^{-1}\sum_{i=1}^{n}\widehat{X}_{i}^{(p,\alpha)}\widehat{Y}_{i}^{(p,\alpha)}$. For convenience, write

[TABLE]

and their empirical conterparts

[TABLE]

By the Cauchy-Schwarz inequality, as $n\rightarrow\infty$,

[TABLE]

and

[TABLE]

Next we deduce a continuous mapping theorem specific for uniform convergence in probability. Suppose $m$ is a continuous $\mathbb{R}\times\mathbb{R}\rightarrow\mathbb{R}$ function. For arbitrary $\varepsilon>0$, there are $w_{n,\varepsilon}\in\{w:\|w\|\leq 1\}$ and $\delta>0$ such that

[TABLE]

which further indicates that

[TABLE]

Lemma 2 follows the identities $\widehat{T}_{p,\alpha}^{*}=\hat{f}_{p,\alpha}^{2}\cdot\hat{g}_{p,\alpha}^{\alpha/(\alpha-1)-1}$ and $T_{p,\alpha}^{*}=f_{p,\alpha}^{2}\cdot g_{p,\alpha}^{\alpha/(\alpha-1)-1}$. ∎

Proof of Proposition 1.

Denote the unit sphere and unit ball in $L^{2}(\mathcal{T})$ by

[TABLE]

respectively. Write

[TABLE]

and

[TABLE]

Clearly, $W_{p-1,\alpha}^{\perp}\cap B$ is weakly sequentially closed and bounded and $T_{\alpha}(w)$ is weakly sequentially upper semi-continuous if constrained on $W_{p-1,\alpha}^{\perp}\cap B$. According to Lemma 1, $T_{\alpha}(w)$ has a maximizer within $W_{p-1,\alpha}^{\perp}\cap B$. This maximizer, say $w^{*}$, must locate in $W_{p-1,\alpha}^{\perp}\cap S$, otherwise we can construct $w^{\prime}=w^{*}/\sqrt{\int_{\mathcal{T}}w^{*2}}$ with $T_{\alpha}(w^{\prime})=\left(\int_{\mathcal{T}}w^{*2}\right)^{\alpha/(\alpha-1)}T_{\alpha}(w^{*})>T_{\alpha}(w^{*})$. Likewise, $\widehat{T}_{\alpha}(w)$ has a maximizer in $\widehat{W}_{p-1,\alpha}^{\perp}\cap S$, too. ∎

Proof of Proposition 2.

Put aside two special cases: when $\alpha=0$, as stated in Section 2.2, $\beta\propto w_{1,0}$; for $\alpha=1/2$, please synthesize (3.4) and (3.11) in [11].

For $p\in\mathbb{N}$ and $\alpha\in(0,1/2)\cup(1/2,1)$, let

[TABLE]

Then $T_{\alpha}=f^{2}\cdot g^{\frac{\alpha}{1-\alpha}-1}$. The Lagrange multiplier rule for Banach spaces [33, pp. 270–271] ensures that there are real numbers $\delta_{1},\ldots,\delta_{p}$, for each $w\in L^{2}(\mathcal{T})$,

[TABLE]

where $\operatorname*{D\!}f(w_{p,\alpha})$, $\operatorname*{D\!}g(w_{p,\alpha})$, $\operatorname*{D\!}h(w_{p,\alpha})$, and $\operatorname*{D\!}e_{j}(w_{p,\alpha})$, all surjections from $L^{2}(\mathcal{T})$ to $\mathbb{R}$, are the first-order (Fréchet) derivatives of $f$, $g$, $h$, and $e_{j}$ evaluated at $w_{p,\alpha}$, respectively; in particular, for $w\in L^{2}(\mathcal{T})$,

[TABLE]

The arbitrariness of $w$ in Eq. (18) entails that

[TABLE]

Cases of $\left(\frac{\alpha}{1-\alpha}-1\right)f^{2}(w_{p,\alpha})g^{\frac{\alpha}{1-\alpha}-1}(w_{p,\alpha})=0$ and $\gamma_{p}=0$ are both eliminated: the former one corresponds to the uninteresting minimum of $T_{\alpha}$, while the latter one leads to the unconstrained maximizer of $T_{\alpha}$ which actually never falls on the unit sphere. By Fredholm’s theorems (see, e.g. [13, 18]), solve the integral equation (19) and acquire

[TABLE]

where $U_{p,\alpha}:L^{2}(\mathcal{T})\rightarrow L^{2}(\mathcal{T})$ takes $w$ to $\left((V_{X}+\gamma_{0}I)^{-1}\circ V_{X}\right)(w)$ with $\gamma_{0}=\gamma_{0}(p,\alpha)\in\mathbb{R}$ and identity operator $I$ and where $\gamma_{1},\ldots,\gamma_{p}$ accommodate the $p$ side-conditions (8). It follows that

[TABLE]

where $K_{p,\alpha}=U_{p,\alpha}\circ\cdots\circ U_{1,\alpha}$, because $w_{p,\alpha}$ is representable in terms of $K_{1,\alpha}(\beta),\ldots,K_{p,\alpha}(\beta)$ for each $p$ and vice versa.

At last we verify that $\beta\in\overline{\text{span}\left\{K_{1,\alpha}(\beta),K_{2,\alpha}(\beta),\ldots\right\}}$. Introduce orthogonal projection operator $P_{p}$ that takes $w\in L^{2}(\mathcal{T})$ to $\sum_{j=1}^{p}\left(\int_{\mathcal{T}}ww_{j,\text{FPC}}\right)w_{j,\text{FPC}}$. Write $\beta_{p,\text{FPC}}=P_{p}(\beta)$. Now,

[TABLE]

in which $\lambda_{j}$ is the $j$-th eigenvalue of $V_{X}$, implying that

[TABLE]

In view of $\left(P_{p}\circ K_{j,\alpha}\right)\left(\beta_{p,\text{FPC}}\right)=\left(P_{p}\circ K_{j,\alpha}\right)(\beta)$ for $1\leq j\leq p$ and $p\in\mathbb{N}$, after taking limits in the $L^{2}$ sense as $p\rightarrow\infty$ on both sides of the following formula

[TABLE]

we accomplish the proof. ∎

Proof of Proposition 3.

For simplicity, we assume that $\lambda_{1}>\lambda_{2}>\cdots>0$ are eigenvalues of operator $V$, i.e., there is no tie among them. Then

[TABLE]

The proposition can be proved by mathematical induction.

For any $w$ ($\neq w_{1,\text{FPC}}$) on $S$ with $\int_{\mathcal{T}}wV_{X}(w)>0$, there exists $\alpha_{0}>2/3$ such that, for all $\alpha\in(\alpha_{0},1)$,

[TABLE]

because $0<\int_{\mathcal{T}}wV_{X}(w)/\lambda_{1}<1$ and ${\operatorname*{cov}}^{2}(Y-\operatorname*{E}Y,\int_{\mathcal{T}}Xw_{1,\text{FPC}})>0$. It follows that $T_{\alpha}(w_{1,\text{FPC}})/T_{\alpha}(w)>1$ for all $\alpha\in(\alpha_{0},1)$ and hence $w_{1,\alpha}=w_{1,\text{FPC}}$ as $\alpha\rightarrow 1$.

Suppose we have $w_{1,\alpha}=w_{1,\text{FPC}},\ldots,w_{p-1,\alpha}=w_{p-1,\text{FPC}}$, for certain $p\geq 2$. For $w$ ($\neq w_{p,\text{FPC}}$) satisfying constraints (8) and $\int_{\mathcal{T}}wV_{X}(w)>0$, along with sufficiently large $\alpha$, the inequalities

[TABLE]

always hold. Thus, as $\alpha\rightarrow 1$, $w_{p,\text{FPC}}=\operatorname*{arg\,max\,}_{w}T_{p,\alpha}(w)$ subject to (8) and hence $w_{p,\alpha}=w_{p,\text{FPC}}$. ∎

Proof of Proposition 4.

Define $S$ and $W_{p-1,\alpha}^{\perp}$ as in the proof of Proposition 1. Apparently, $T_{\alpha}(w)=T_{p,\alpha}^{*}(w)$ for all $w\in W_{p-1,\alpha}^{\perp}$. That is, $w_{p,\alpha}$ is also the solution to

[TABLE]

For any $w\in S$, construct $w^{*}\in S$ proportional to

[TABLE]

Due to

[TABLE]

and $\alpha/(\alpha-1)<0$ (excluding the trivial case $\alpha=0$), it is easy to verify that $w^{*}\in W_{p-1,\alpha}^{\perp}$ and

[TABLE]

The inequality is an equality only when $w\in W_{p-1,\alpha}^{\perp}$, In other words, it suffices to drop side-conditions (20) when maximizing $T_{p,\alpha}^{*}(w)$ subject to $\|w\|=1$. ∎

Proof of Proposition 5.

Change every population values in the proof of Proposition 4 into empirical counterparts. ∎

Proof of Proposition 6.

For $p\in\mathbb{N}$, let $h=h(w)=\int_{\mathcal{T}}w^{2}$,

[TABLE]

and

[TABLE]

Then $T_{p,\alpha}=f_{p,\alpha}^{2}\cdot g_{p,\alpha}^{\frac{\alpha}{1-\alpha}-1}$ and $w_{p,\alpha}$ defined as (7) must be a solution to the constrained optimization problem

[TABLE]

for certain $g_{0}\in(0,\lambda_{1}^{(p,\alpha)}]$, where $\lambda_{j}^{(p,\alpha)}$ is the $j$-th largest eigenvalue of operator $V_{X^{(p,\alpha)}}$ with corresponding eigenfunction $\phi_{j}^{(p,\alpha)}$.

Check the case with $g_{0}=\lambda_{1}^{(p,\alpha)}>0$ (i.e., the functional principal component basis). Provided that $\lambda_{1}^{(p,\alpha)}$ has multiplicity $=m\geq 1$, we can write $w_{p,\alpha}=\sum_{j=1}^{m}a_{j}\phi_{j}^{(p,\alpha)}$, where $a_{1},\ldots,a_{m}\in[-1,1]$ and $\sum_{j=1}^{m}a_{j}^{2}=1$. The Cauchy-Schwarz inequality implies that the maximum of

[TABLE]

is achieved if and only if

[TABLE]

Therefore,

[TABLE]

Unless $g_{0}=\lambda_{1}^{(p,\alpha)}>0$, apply the Lagrange multiplier rule for Banach spaces as in the proof of Proposition 2 and arrive at

[TABLE]

with $\delta_{1},\delta_{2}\in\mathbb{R}$. $\delta_{2}$ must be nonzero as the solution to the unconstraint optimization problem $\max T_{p,\alpha}^{*}$ never falls on the unit sphere. Also, we rule out the case $f_{p,\alpha}(w_{p,\alpha})=0$ which corresponds to the uninteresting minimum of $T_{p,\alpha}^{*}$.

If $\delta_{1}=0$, the functional continuum basis reduces to functional PLS basis and $w_{p,\alpha}\propto V_{X^{(p,\alpha)}}(\beta)$. When $\delta^{(p,\alpha)}$ is close enough to [math], $\lambda_{1}^{(p,\alpha)}/\delta^{(p,\alpha)}$ becomes dominant over $\lambda_{j}^{(p,\alpha)}$ for all $j$, i.e., $\lambda_{j}^{(p,\alpha)}+\lambda_{1}^{(p,\alpha)}/\delta^{(p,\alpha)}$ and $\lambda_{j^{\prime}}^{(p,\alpha)}+\lambda_{1}^{(p,\alpha)}/\delta^{(p,\alpha)}$ approach each other for all $j\neq j^{\prime}$. Accordingly,

[TABLE]

In the case with nonzero $\delta_{1}$, solving the following inhomogeneous Fredholm integral equation with respect to $w_{p,\alpha}$,

[TABLE]

we also obtain the solution

[TABLE]

where $\delta^{(p,\alpha)}=\delta_{1}\lambda_{1}^{(p,\alpha)}/\delta_{2}$. The existence and uniqueness of this solution is guaranteed by Fredholm’s theorems which hold here because $v_{X^{(p,\alpha)}}\in L^{2}(\mathcal{T}\times\mathcal{T})$.

The last phase of this proof is to ascertain that $\delta^{(p,\alpha)}\notin(-\infty,-1)$. Without loss of generality, assume that $\int_{\mathcal{T}}\phi_{j}^{(p,\alpha)}V_{X^{(p,\alpha)}}(\beta)\geq 0$ for all $j$, otherwise we can use $-\phi_{j}^{(p,\alpha)}$ instead. $\sum_{j=1}^{\infty}\lambda_{j}^{(p,\alpha)}<\infty$ is a property of Hilbert-Schmidt operator $v_{X^{(p,\alpha)}}$, further indicating that, if $\delta^{(p,\alpha)}\in(-\infty,-1)$, then there must exist $j_{0}$ such that $\lambda_{j_{0}}^{(p,\alpha)}+\lambda_{1}^{(p,\alpha)}/\delta^{(p,\alpha)}$ is negative. Under this circumstance, changing the sign of it will increase $f_{p,\alpha}^{2}(w_{p,\alpha})$ without altering $g_{p,\alpha}(w_{p,\alpha})$ or violating the unit norm constraint. This contradicts the definition of $w_{p,\alpha}$ and completes the proof. ∎

Proof of Theorem 1.

We resort to an argument similar to the proof adopted by Amemiya [2, Theorem 4.1.1] and extend it from the finite-dimensional setting to the functional context. The unit ball $B$ is as defined in the proof of Proposition 1. Start with $p=1$ and let $N$ be a neighborhood in $L^{2}(\mathcal{T})$ containing $w_{p,\alpha}$, namely,

[TABLE]

Verify that $B\setminus N_{1,\delta}$ is weakly sequentially closed and bounded and $T_{1,\alpha}^{*}(w)$ is weakly sequentially upper semi-continuous within $B\setminus N_{1,\delta}$. Then Lemma 1 guarantees the existence of $\max_{w\in B\setminus N_{1,\delta}}T_{1,\alpha}^{*}(w)$.

Write

[TABLE]

and observe that

[TABLE]

Implied by Lemma 2, $\lim_{n\rightarrow\infty}\Pr\left\{\hat{w}_{1,\alpha}\in N_{1,\delta}\right\}=1$. Considering the arbitrariness of $\delta$, we conclude that $\hat{w}_{1,\alpha}\overset{\text{P}}{\longrightarrow}w_{1,\alpha}$ as $n\rightarrow\infty$. In case the convergence of $\hat{w}_{1,\alpha},\ldots,\hat{w}_{p-1,\alpha}$ holds, the prerequisite of Lemma 2 is fulfilled. Mimicking the argument for $p=1$, we deduce $\hat{w}_{p,\alpha}\overset{\text{P}}{\longrightarrow}w_{p,\alpha}$ as $n\rightarrow\infty$.

As for $\hat{\beta}_{p,\alpha}$ and $\hat{\eta}_{p,\alpha}(x)$, the convergence can be proved after we recall their definitions in (11) and (12) and employ the continuous mapping theorem for convergence in probability. ∎

Proof of Proposition 7.

Follow the identical argument in the proof for Proposition 6 but substitute empirical items for the population counterparts. Meanwhile, take the following identity into consideration:

[TABLE]

∎