The adaptive Wynn-algorithm in generalized linear models with univariate response

TL;DR

This paper introduces an adaptive Wynn-algorithm for D-optimal design in generalized linear models with univariate responses, demonstrating its strong consistency and asymptotic properties under certain conditions.

Contribution

It extends the Wynn-algorithm to generalized linear models, providing theoretical guarantees for adaptive design and estimator asymptotics.

Findings

Adaptive ML-estimators are strongly consistent.

Design sequence is asymptotically locally D-optimal.

Asymptotic normality holds under smoothness assumptions.

Abstract

For a nonlinear regression model the information matrices of designs depend on the parameter of the model. The adaptive Wynn-algorithm for D-optimal design estimates the parameter at each step on the basis of the employed design points and observed responses so far, and selects the next design point as in the classical Wynn-algorithm for D-optimal design. The name `Wynn-algorithm' is in honor of Henry P. Wynn who established the latter `classical' algorithm in his 1970 paper. The asymptotics of the sequences of designs and maximum likelihood estimates generated by the adaptive algorithm is studied for an important class of nonlinear regression models: generalized linear models whose (univariate) response variables follow a distribution from a one-parameter exponential family. Under the assumptions of compactness of the experimental region and of the parameter space together with some natural continuity assumptions it is shown that the adaptive ML-estimators are strongly consistent and the design sequence is asymptotically locally D-optimal at the true parameter point. If the true parameter point is an interior point of the parameter space then under some smoothness assumptions the asymptotic normality of the adaptive ML-estimators is obtained.

Equations

$$\xi =\sum _{x\in \mathrm{supp}(\xi )}\xi (x){\delta }_x,$$

$$M(\xi ,\theta )=\sum _{x\in \mathrm{supp}(\xi )}\xi (x)f_{\theta }(x)f_{\theta }^{\mathsf{T}}(x),$$

$$\psi(x,\theta)\,=\,\varphi\bigl{(}f^{\sf\scriptsize T}(x)\,\theta\bigr{)},\ \ x\in{\cal X},\ \theta\in\Theta,$$

$$\displaystyle\mu(x,\theta)\,=\,\exp\bigl{(}f^{\sf\scriptsize T}(x)\,\theta\bigr{)}\Big{/}\Bigl{(}1+\exp\bigl{(}f^{\sf\scriptsize T}(x)\,\theta\bigr{)}\Bigr{)},\ \ x\in{\cal X},\ \theta\in\Theta,$$

$$\displaystyle\mbox{and }\ \varphi(u)\,=\,\exp(u/2)\,\big{/}\,\bigl{(}1+\exp(u)\bigr{)},\ \ u\in\mathbb{R}.$$

$$\displaystyle\xi_{n}\,=\,\frac{1}{n}\sum_{i=1}^{n}\delta_{x_{i}}\quad\mbox{for all $n\geq n_{\rm\scriptsize st}$,}$$

$$\displaystyle x_{n+1}\,=\,\arg\max_{x\in{\cal X}}f_{\theta_{n}}^{\sf\scriptsize T}(x)M^{-1}(\xi_{n},\theta_{n})f_{\theta_{n}}(x)\quad\mbox{for all $n\geq n_{\rm\scriptsize st}$,}$$

$${\xi }_{n+1}$$

$$\displaystyle M\bigl{(}\xi_{n+1},\theta\bigr{)}$$

$$\inf_{n\geq n_{\rm\tiny st}}\lambda_{\scriptsize\rm min}\bigl{(}M(\xi_{n},\theta_{n})\bigr{)}\ >0,$$

$$\inf_{n\geq n_{\rm\tiny st},\,\theta\in\Theta}\lambda_{\scriptsize\rm min}\bigl{(}M(\xi_{n},\theta)\bigr{)}\ >0,$$

$${\beta }_n>\beta$$

$${\beta }_n\le \beta$$

$$n_1\ge m_0$$

$$n_2\ge m_0$$

$$\gamma$$

$$\kappa$$

$$f_{{\theta }_0}^{\mathsf{T}}(x_0)M^{-1}({\xi }_0,{\theta }_0)f_{{\theta }_0}(x_0)\le \frac{1}{(1-\eta {)}^2}{a}^{\mathsf{T}}{M}^{-1}({\xi }_0,{\theta }_0)a.$$

$$\|b_{0}\|^{2}\,=\,\frac{f_{\theta_{0}}^{\sf\scriptsize T}(x_{0})M^{-2}_{0}f_{\theta_{0}}(x_{0})}{\bigl{(}f_{\theta_{0}}^{\sf\scriptsize T}(x_{0})M^{-1}_{0}f_{\theta_{0}}(x_{0})\bigr{)}^{2}}.$$

$$f_{{\theta }_0}^{\mathsf{T}}(x_0)M_0^{-2}{f}_{{\theta }_0}(x_0)$$

$$f_{{\theta }_0}^{\mathsf{T}}(x_0)M_0^{-1}{f}_{{\theta }_0}(x_0)$$

$$f_{{\theta }_0}^{\mathsf{T}}(x_0)M_0^{-2}{f}_{{\theta }_0}(x_0)\le {\lambda }_1^{-2}{f}_{{\theta }_0}^{\mathsf{T}}(x_0)f_{{\theta }_0}(x_0)={\lambda }_1^{-2}\Vert f_{{\theta }_0}(x_0){\Vert }^2\le {\lambda }_1^{-2}{\gamma }^2.$$

$$\kappa\,\leq\max_{x\in{\cal X}}\bigl{\{}\bigl{(}v_{1}^{\sf\scriptsize T}f_{\theta_{0}}(x)\bigr{)}^{2}\bigr{\}}\,=\,\bigl{(}v_{1}^{\sf\scriptsize T}f_{\theta_{0}}(z_{0})\bigr{)}^{2}$$

$$f_{{\theta }_0}^{\mathsf{T}}(x_0)M_0^{-1}{f}_{{\theta }_0}(x_0)\ge f_{{\theta }_0}^{\mathsf{T}}(z_0)M_0^{-1}{f}_{{\theta }_0}(z_0)\ge {\lambda }_1^{-1}{f}_{{\theta }_0}^{\mathsf{T}}(z_0)v_1{v}_1^{\mathsf{T}}{f}_{{\theta }_0}(z_0)$$

$$\displaystyle\,=\,\lambda_{1}^{-1}\,\bigl{(}v_{1}^{\sf\scriptsize T}f_{\theta_{0}}(z_{0})\bigr{)}^{2}\geq\,\lambda_{1}^{-1}\,\kappa\ .$$

$$\Vert b_0\Vert \le \gamma /\kappa .$$

$$\frac{1}{b_0^{\mathsf{T}}{M}_0{b}_0}=f_{{\theta }_0}^{\mathsf{T}}(x_0)M_0^{-1}{f}_{{\theta }_0}(x_0)\text{mbox}and{f}_{{\theta }_0}^{\mathsf{T}}(x_0)b_0=1.$$

$$|a^{\sf\scriptsize T}b_{0}-1|=|a^{\sf\scriptsize T}b_{0}-f_{\theta_{0}}^{\sf\scriptsize T}(x_{0})\,b_{0}|=\big{|}\bigl{(}a-f_{\theta_{0}}(x_{0})\bigr{)}^{\sf\scriptsize T}b_{0}\big{|}\leq\|a-f_{\theta_{0}}(x_{0})\|\,\|b_{0}\|\,\leq\,\eta,$$

$$a^{\mathsf{T}}{M}_0^{-1}a\ge \frac{1}{b^{\mathsf{T}}{M}_{0}b}=\frac{(a^{\mathsf{T}}{b}_0{)}^2}{b_0^{\mathsf{T}}{M}_0{b}_0}$$

$$=(a^{\mathsf{T}}{b}_0{)}^2{f}_{{\theta }_0}^{\mathsf{T}}(x_0)M_0^{-1}{f}_{{\theta }_0}(x_0)\ge (1-\eta {)}^2{f}_{{\theta }_0}^{\mathsf{T}}(x_0)M_0^{-1}{f}_{{\theta }_0}(x_0),$$

$$\|f_{\theta_{n}}(x)-f_{\theta_{n}}(z)\|\,\leq\,\eta\kappa/\gamma\ \ \mbox{for all $x,z\in S$, and }\ \xi_{n}(S)>\frac{1}{(1-\eta)^{2}p}.$$

$${\overline{f}}_{{\theta }_n}(S,{\xi }_n):=\frac{1}{{\xi }_n(S)}\sum _{x\in S\cap \mathrm{supp}({\xi }_n)}{\xi }_n(x)f_{{\theta }_n}(x).$$

$$f_{{\theta }_n}^{\mathsf{T}}(x_{n+1})M^{-1}({\xi }_n,{\theta }_n)f_{{\theta }_n}(x_{n+1})\le \frac{1}{(1-\eta {)}^2}{\overline{f}}_{{\theta }_n}^{\mathsf{T}}(S,{\xi }_n)M^{-1}({\xi }_n,{\theta }_n){\overline{f}}_{{\theta }_n}(S,{\xi }_n).$$

$${\overline{f}}_{{\theta }_n}^{\mathsf{T}}(S,{\xi }_n)M^{-1}({\xi }_n,{\theta }_n){\overline{f}}_{{\theta }_n}(S,{\xi }_n)$$

Full text

1 Introduction

In a nonlinear regression model the information matrix of a design depends on the model parameter $\theta\in\Theta$ whose true value is unknown. Modifying the classical algorithm of Wynn [16] for sequential generation of a D-optimal design in linear regression to an adaptive sequential procedure in a nonlinear model, the ‘adaptive Wynn-algorithm’ emerges, which was called ‘one-step ahead adaptive D-optimal design algorithm’ in Pronzato [11].

By $\mathbb{N}$, $\mathbb{N}_{0}$, $\mathbb{R}$, and $\mathbb{R}^{p}$ we denote the set of all positive integers, the set of all nonnegative integers, the real line, and the $p$-dimensional Euclidean space, respectively. Vectors $a\in\mathbb{R}^{p}$ are written as column vectors and $a^{\sf\scriptsize T}$ denotes the transposed of $a$, which is a $p$-dimensional row vector. The usual Euclidean norm on $\mathbb{R}^{p}$ is denoted by $\|a\|=(a^{\sf\scriptsize T}a)^{1/2}$. If $(a_{i})_{i\in I}$ is a family of vectors $a_{i}\in\mathbb{R}^{p}$ then {\rm span}\bigl{\{}a_{i}\,:\,i\in I\bigr{\}} denotes the linear subspace of $\mathbb{R}^{p}$ generated by the vectors $a_{i}$ ($i\in I$). For a linear subspace $V$ of $\mathbb{R}^{p}$ the dimension of $V$ is denoted by ${\rm dim}(V)$. If $A$ is a symmetric $p\times p$ matrix then ${\rm tr}(A)$ denotes the trace of $A$ and $\|A\|$ denotes the Frobenius norm of $A$, i.e., \|A\|=\bigl{(}{\rm tr}(A^{2})\bigr{)}^{1/2}. For any two symmetric $p\times p$ matrices $A$ and $B$ we write $A\leq B$ or, equivalently, $B\geq A$ iff $B-A$ is nonnegative definite. Thereby a semi-ordering is defined on the set of all symmetric $p\times p$ matrices, which is called the Loewner semi-ordering.

We give an outline of the adaptive Wynn-algorithm. Let ${\cal X}$ be the experimental region and $\Theta$ be the parameter space. For each $\theta\in\Theta$ a function  $f_{\theta}\,:\,{\cal X}\longrightarrow\mathbb{R}^{p}$ is given such that the range of $f_{\theta}$ spans $\mathbb{R}^{p}$, i.e., {\rm span}\bigl{\{}f_{\theta}(x)\,:\,x\in{\cal X}\bigr{\}}=\mathbb{R}^{p} for each $\theta\in\Theta$. Throughout it is assumed that ${\cal X}$ and $\Theta$ are compact metric spaces with distance functions $d_{{\cal X}}$ and $d_{\Theta}$, resp., and the function $(x,\theta)\mapsto f_{\theta}(x)$ is continuous on ${\cal X}\times\Theta$. Of course, the assumption of compactness of the parameter space $\Theta$ is somewhat disturbing but, presently, indispensable for our results. However, in the literature on adaptive procedures in stochastic approximation it is quite common to assume compactness of the parameter space and, moreover, to assume the true parameter to be an interior point, see e.g. Venter [14], Section 4.

An (approximate) design $\xi$ is a probability measure with finite support on ${\cal X}$, and it can formally be represented as

[TABLE]

where ${\rm supp}(\xi)$ denotes the support of $\xi$, which is a nonempty finite subset of ${\cal X}$, and to each $x\in{\rm supp}(\xi)$ the design $\xi$ assigns a positive weight $\xi(x)>0$ such that $\sum_{x\in{\scriptsize\rm supp}(\xi)}\xi(x)=1$. The symbol $\delta_{x}$ (for any $x\in{\cal X}$) stands for the one-point probability measure on ${\cal X}$ concentrated at the point $x$. For a design $\xi$ and for a parameter point $\theta\in\Theta$ the information matrix (per observation) of $\xi$ at $\theta$ is given by

[TABLE]

which is a nonnegative definite $p\times p$ matrix. The information matrices defined by (1.1) arise as Fisher information in some nonlinear regression model and, in particular, the functions $f_{\theta}$ are related to a local linearization at $\theta$ of the (univariate) nonlinear mean response $\mu(x,\theta)$, say. E.g., in case of a homoscedastic regression model the vector $f_{\theta}(x)$ is given by the gradient of $\mu(x,\,\cdot\,)$ at $\theta$. In case of heteroscedasticity, also the variance function and possibly its gradient enters into $f_{\theta}$, see Atkinson et al. [1]. For the case of a generalized linear model the functions $f_{\theta}$ have the pleasant property that the parameter $\theta$ only enters into a positive scalar factor, i.e., a real-valued positive function $\psi(x,\theta)$ while the ‘body’ of the functions $f_{\theta}$ is given by one $\mathbb{R}^{p}$-valued function $f$. We will refer to this situation as ‘condition (GLM)‘ on the family of functions $f_{\theta}$, $\theta\in\Theta$, namely:

**Condition (GLM)

$f_{\theta}(x)\,=\,\psi(x,\theta)\,f(x)$**  for all $(x,\theta)\in{\cal X}\times\Theta$, where $\psi\,:\,{\cal X}\times\Theta\longrightarrow(\,0\,,\,\infty)$  and  $f:\,{\cal X}\longrightarrow\mathbb{R}^{p}$  are given continuous functions. $\Box$

For a generalized linear model one has, even more specially, that $\Theta\subseteq\mathbb{R}^{p}$ and the real-valued function $\psi$ is actually a function of $f^{\sf\scriptsize T}(x)\,\theta$, i.e.,

[TABLE]

where $\varphi$ is a continuous function of one real variable. As an example, for the logistic model with Bernoulli response variables one has

[TABLE]

see Atkinson and Woods [2], Section 2.3.

The adaptive Wynn algorithm generates a sequence of designs $\xi_{n}$, $n\geq n_{\rm\scriptsize st}$, (the index ‘st’ standing for ‘starting’) which is obtained from a sequence of points $x_{i}\in{\cal X}$, $i\in\mathbb{N}$, and a sequence of parameter points $\theta_{n}\in\Theta$, $n\geq n_{\rm\scriptsize st}$, as follows,

[TABLE]

where it is assumed that the starting design $\xi_{n_{\rm\tiny st}}$ is such that its information matrix $M(\xi_{n_{\rm\tiny st}},\theta)$ is positive definite for all $\theta\in\Theta$. This implies positive definiteness of the information matrices of all designs $\xi_{n}$, $n\geq n_{\rm\scriptsize st}$, since

[TABLE]

which entails  $M(\xi_{n},\theta)\geq(n_{\rm\scriptsize st}/n)\,M(\xi_{n_{\rm\tiny st}},\theta)$, for all $n\geq n_{\rm\scriptsize st}$ and all $\theta\in\Theta$. Note that the design $\xi_{n}$ for each $n\geq n_{\rm\scriptsize st}$ is an exact design of size $n$ since the weights assigned to its support points are integer multiples of $1/n$, and hence $\xi_{n}$ can be exactly realized for the sample size $n$. The sequence of parameter points $\theta_{n}\in\Theta$, $n\geq n_{\rm\scriptsize st}$, employed will actually be generated by adaptive parameter estimation, i.e., $\theta_{n}=\widehat{\theta}_{n}(x_{1},y_{1},\ldots,x_{n},y_{n})$ for all $n\geq n_{\rm\scriptsize st}$, where $y_{1},\ldots y_{n},\ldots$ are the sequentially observed univariate responses at the design points $x_{1},\ldots,x_{n},\ldots$, resp., due to an underlying regression model with a mean response function $\mu(x,\theta)$, $x\in{\cal X}$, $\theta\in\Theta$, as mentioned above.

In Section 2 we study the asymptotic behavior of the design sequence $\xi_{n}$ and their information matrices under any sequence of parameter points $\theta_{n}$, $n\geq n_{\rm\scriptsize st}$, which may be thought of as a path of a sequence of adaptive estimators $\widehat{\theta}_{n}$, $n\geq n_{\rm\scriptsize st}$. Also, the design sequence $\xi_{n}$, $n\geq n_{\rm\scriptsize st}$, may be viewed as a path of a sequence of adaptive random designs. In Section 3 the asymptotic properties (strong consistency, asymptotic normality) of adaptive ML-estimators in the algorithm are derived. For modelling the adaptive procedure inherent in the algorithm we follow the martingale approach of Lai and Wei [10], Lai [9], and Chen, Hu and Ying [4]. Some known results on matrices used in our proofs are collected in the appendix.

The paper of Pronzato [11] deals with the adaptive Wynn-algorithm for the case of a finite design space (and a compact parameter space). In that paper, under some conditions of Chebyshev type on the functions $f_{\theta}$, $\theta\in\Theta$, and the mean response function, asymptotic results of the design sequence and of adaptive least squares estimators were derived, and also for adaptive ML-estimators in the particular case of binary response variables. The thesis of Freise [6] provides an interesting contribution to the asymptotics of the adaptive Wynn algorithm. Of further interest, though not dealing with adaptive procedures, are the papers of Wu [15] on nonlinear least squares estimators, and of Fahrmeir and Kaufmann [5] on maximum likelihood estimators in generalized linear models.

2 Asymptotic properties of designs

Throughout this section let $\theta_{n}\in\Theta$, $n\geq n_{\rm\scriptsize st}$, be any given sequence of parameter points and let $\xi_{n}$, $n\geq n_{\rm\scriptsize st}$, be the sequence of designs given by (1.3) and (1.4), where the starting design $\xi_{n_{\rm\tiny st}}$ is such that its information matrix $M(\xi_{n_{\rm\tiny st}},\theta)$ is positive definite for all $\theta\in\Theta$, and hence $M(\xi_{n},\theta)$ is positive definite for all $n\geq n_{\rm\scriptsize st}$ and all $\theta\in\Theta$.

An important question is whether positive definiteness of the information matrices of the designs $\xi_{n}$ is preserved asymptotically in the sense that

[TABLE]

or, even stronger,

[TABLE]

where $\lambda_{\rm\scriptsize min}(A)$ denotes the smallest eigenvalue of a symmetric matrix $A$. Answers to the questions about asymptotic nonsingularity will be given. Under condition (GLM) the stronger asymptotic nonsingularity (2.2) holds true, while a weaker technical condition (T) ensures the asymptotic nonsingularity (2.1). We start our derivations with four lemmas.

For a real number $a$ we denote by   $\lceil a\rceil$  the smallest integer greater than or equal to $a$.

Lemma 2.1

Let $\beta_{n}$, $n\geq m_{0}$, be a sequence in $[\,0\,,\,1]$, where $m_{0}\in\mathbb{N}$ is given, and let $\beta\in(\,0\,,\,1)$ such that for each $n\geq m_{0}$ the following two implications hold.

[TABLE]

*Let $\widetilde{\beta}>\beta$ be given. Denote  m_{1}\,=\,m_{1}(\beta,\widetilde{\beta},m_{0})\,:=\,\big{\lceil}1/\beta\big{\rceil}\,\max\bigl{\{}m_{0},\,\big{\lceil}1/(\widetilde{\beta}-\beta)\big{\rceil}\bigr{\}}.

Then:   $\beta_{n}\,\leq\,\widetilde{\beta}$  for all $n\geq m_{1}$.*

Proof.  We show that

[TABLE]

Let $n_{1}\geq m_{0}$ with $\beta_{n_{1}}\leq\beta$ be given. In case that the sequence $\beta_{n}$, $n\geq n_{1}$, never exceeds $\beta$ the conclusion in (2.5) trivially holds. In the other case, by (2.3), it suffices to show that $\beta_{n}\leq\beta+\frac{1}{n_{1}}$ holds for those $n>n_{1}$ for which $\beta_{n-1}\leq\beta$ and $\beta_{n}>\beta$. For such $n$, by (2.4), $\beta_{n}\leq\beta_{n-1}+\frac{1}{n}\leq\beta+\frac{1}{n_{1}}$.

Next we show that

[TABLE]

Let $n_{2}\geq m_{0}$ with $\beta_{n_{2}}>\beta$ be given. If $r$ is a nonnegative integer such that $\beta_{n_{2}+k}>\beta$ for all $k=0,\ldots,r$, then by (2.3)  $\beta<\beta_{n_{2}+r}=\frac{n_{2}}{n_{2}+r}\beta_{n_{2}}\leq\frac{n_{2}}{n_{2}+r}$ and hence $\beta<\frac{n_{2}}{n_{2}+r}$, i.e., $n_{2}+r<n_{2}/\beta$. So there must be some $\nu\in\{n_{2}+1,\ldots,\lceil n_{2}/\beta\rceil\}$ such that $\beta_{\nu}\leq\beta$.

Consider $m_{1}$ as defined in the lemma and define $k_{1}=\max\{m_{0},\lceil 1/(\widetilde{\beta}-\beta)\rceil\}$. Note that m_{1}=\big{\lceil}1/\beta\big{\rceil}\,k_{1}. We show that $\beta_{n}\leq\widetilde{\beta}$ for all $n\geq m_{1}$.

Case 1: $\beta_{k_{1}}\leq\beta$. By (2.5) with $n_{1}=k_{1}$ one gets $\beta_{n}\leq\beta+\frac{1}{k_{1}}\leq\widetilde{\beta}$ for all $n\geq m_{1}$.

Case 2: $\beta_{k_{1}}>\beta$. By (2.6) with $n_{2}=k_{1}$ one gets some $\nu\in\{k_{1}+1,\ldots,\lceil k_{1}/\beta\rceil\}$ such that $\beta_{\nu}\leq\beta$. Application of (2.5) on $n_{1}=\nu$ yields $\beta_{n}\leq\beta+\frac{1}{\nu}\leq\widetilde{\beta}$ for all $n\geq\nu$ and, in particular, $\beta_{n}\leq\widetilde{\beta}$ for all $n\geq m_{1}$ since \nu\leq\big{\lceil}k_{1}/\beta\big{\rceil}\leq\lceil 1/\beta\rceil\,k_{1}=m_{1}. $\Box$

We will use the two positive real constants given by

[TABLE]

where in (2.8) the infimum is taken over all $v$ from the unit sphere of $\mathbb{R}^{p}$ and over all $\theta\in\Theta$. In fact, both the supremum in (2.7) and the infimum in (2.8) are attained and are positive. This is obvious for the former supremum by the continuity and compactness assumptions. For the infimum in (2.8), note that the function (v,\theta)\mapsto\max_{x\in{\cal X}}\bigl{(}v^{\sf\scriptsize T}f_{\theta}(x)\bigr{)}^{2} is lower semi-continuous (as a pointwise maximum of a family of continuous functions) and positive, where the latter follows from the basic assumption that the image $\{f_{\theta}(x)\,:\,x\in{\cal X}\}$ spans $\mathbb{R}^{p}$ for each $\theta\in\Theta$. By compactness of the unit sphere of $\mathbb{R}^{p}$ and compactness of $\Theta$ the infimum in (2.8) is attained and hence positive.

Lemma 2.2

*Let $\xi_{0}$ be a design and $\theta_{0}\in\Theta$ such that $M(\xi_{0},\theta_{0})$ is positive definite.

Let $x_{0}\,=\,\arg\max_{x\in{\cal X}}f_{\theta_{0}}^{\sf\scriptsize T}(x)M^{-1}(\xi_{0},\theta_{0})f_{\theta_{0}}(x)$ and $\eta\in(\,0\,,\,1\,)$. Then for all $a\in\mathbb{R}^{p}$ such that  $\|f_{\theta_{0}}(x_{0})-a\|\leq\eta\kappa/\gamma$ one has*

[TABLE]

Proof.  Abbreviate $M_{0}=M(\xi_{0},\theta_{0})$. Define  b_{0}\,:=\,M^{-1}_{0}f_{\theta_{0}}(x_{0})\big{/}\bigl{(}f_{\theta_{0}}^{\sf\scriptsize T}(x_{0})M^{-1}_{0}f_{\theta_{0}}(x_{0})\bigr{)}.  Then

[TABLE]

Denote by $\lambda_{1}$ the smallest eigenvalue of $M_{0}$. We will show that

[TABLE]

To prove (2.10), the obvious inequality (in the Loewner semi-ordering)  $M^{-2}_{0}\leq\lambda_{1}^{-2}\,I_{p}$, where $I_{p}$ denotes the $p\times p$ unit matrix, yields

[TABLE]

To prove (2.11) let $v_{1}$ be a normalized eigenvector to $\lambda_{1}$ of $M_{0}$. By definition of $\kappa$ from (2.8)

[TABLE]

for some $z_{0}\in{\cal X}$. Hence, together with the obvious inequality (in the Loewner semi-ordering)  $M^{-1}_{0}\geq\lambda_{1}^{-1}\,v_{1}v_{1}^{\sf\scriptsize T}$, one obtains

[TABLE]

From (2.9), (2.10), and (2.11) we get

[TABLE]

Let $a\in\mathbb{R}^{p}$ such that $\|f_{\theta_{0}}(x_{0})-a\|\leq\eta\kappa/\gamma$. Recall that, by definition of $b_{0}$,

[TABLE]

Together with (2.12) we get

[TABLE]

hence $a^{\sf\scriptsize T}b_{0}\geq 1-\eta>0$. Define $b:=b_{0}/(a^{\sf\scriptsize T}b_{0})$. Then $a^{\sf\scriptsize T}b=1$ and hence by (M3) of the appendix,

[TABLE]

from which the result follows. $\Box$

Lemma 2.3

Let $p\geq 2$ and \eta\in\bigl{(}\,0\,,\,1-\frac{1}{\sqrt{p}}\,\bigr{)}. Let $S\subseteq{\cal X}$ and $n\geq n_{\rm\scriptsize st}$ be given such that

[TABLE]

Then:  $x_{n+1}\not\in S$.

Proof.  Suppose that $x_{n+1}\in S$. Consider the mean (of $f_{\theta_{n}}(x)$ over $S$ w.r.t. $\xi_{n}$),

[TABLE]

Since $\|f_{\theta_{n}}(x_{n+1})-f_{\theta_{n}}(x)\|\leq\eta\kappa/\gamma$ for all $x\in S$ we get $\|f_{\theta_{n}}(x_{n+1})-\overline{f}_{\theta_{n}}(S,\xi_{n})\|\leq\eta\kappa/\gamma$. By Lemma 2.2,

[TABLE]

By (M1) and (M2) of the appendix,

[TABLE]

Hence it follows that $f_{\theta_{n}}^{\sf\scriptsize T}(x_{n+1})\,M^{-1}(\xi_{n},\theta_{n})\,f_{\theta_{n}}(x_{n+1})\,<\,p$. This is a contradiction since we know from the Kiefer-Wolfowitz equivalence Theorem that $\max_{x\in{\cal X}}f_{\theta_{n}}^{\sf\scriptsize T}(x)\,M^{-1}(\xi_{n},\theta_{n})\,f_{\theta_{n}}(x)\,\geq\,p$. So $x_{n+1}\not\in S$ must be true. $\Box$

For $a\in\mathbb{R}^{p}$, $\emptyset\not=V\subseteq\mathbb{R}^{p}$, and $\varepsilon>0$ we denote

[TABLE]

The set $\overline{V}(\varepsilon)$ may be called an $\varepsilon$-neighborhood of $V$. For any subset $C\subseteq\mathbb{R}^{p}$ and $\theta\in\Theta$ we denote, as usual, f_{\theta}^{-1}(C)\,=\,\bigl{\{}x\in{\cal X}\,:\,f_{\theta}(x)\in C\bigr{\}}.

Lemma 2.4

Let $V\subseteq\mathbb{R}^{p}$ be a linear subspace with ${\rm dim}(V)\leq p-1$, and let $\delta$ with $0<\delta\leq\sqrt{\kappa}$ and $n\geq n_{\rm\scriptsize st}$ be given. Then, denoting  w_{n}:=\xi_{n}\Bigl{(}f_{\theta_{n}}^{-1}\bigl{(}\overline{V}(\delta)\bigr{)}\Bigr{)}, one has

[TABLE]

Proof.  For all $x\in{\cal X}$ decompose

[TABLE]

where $V^{\perp}$ denotes the orthogonal complement of $V$ in $\mathbb{R}^{p}$. Choose  $x^{*}\,=\,\arg\max_{x\in{\cal X}}\|v(x)\|$.  Clearly, for all x\in f_{\theta_{n}}^{-1}\bigl{(}\overline{V}(\delta)\bigr{)} one has \|v(x)\|={\rm dist}\bigl{(}f_{\theta_{n}}(x),V\bigr{)}\leq\delta. On the other hand, $\|v(x^{*})\|\geq\sqrt{\kappa}$ which can be seen as follows. Since ${\rm dim}(V)\leq p-1$ there is some $(p-1)$-dimensional linear subspace $W\subseteq\mathbb{R}^{p}$ such that $V\subseteq W$. There is a representation $W=\{a\in\mathbb{R}^{p}\,:\,c^{\sf\scriptsize T}a=0\}$ for some $c\in\mathbb{R}^{p}$ with $\|c\|=1$. By ${\rm dist}(a,V)\geq{\rm dist}(a,W)$ for all $a\in\mathbb{R}^{p}$, and by definition of $\kappa$ in (2.8) one gets

[TABLE]

Define $b\,:=\,v(x^{*})/\|v(x^{*})\|^{2}$. Clearly, $b^{\sf\scriptsize T}f_{\theta_{n}}(x^{*})=1$ hence by (M3) of the appendix

[TABLE]

Now, \displaystyle b^{\sf\scriptsize T}M(\xi_{n},\theta_{n})\,b\,=\,\sum_{x\in{\rm\scriptsize supp}(\xi_{n})}\xi_{n}(x)\,\bigl{(}b^{\sf\scriptsize T}f_{\theta_{n}}(x)\bigr{)}^{2}\,=\,\sum_{x\in{\rm\scriptsize supp}(\xi_{n})}\xi_{n}(x)\,\frac{\bigl{(}v^{\sf\scriptsize T}(x^{*})\,v(x)\bigr{)}^{2}}{\|v(x^{*})\|^{4}},

and \bigl{(}v^{\sf\scriptsize T}(x^{*})\,v(x)\bigr{)}^{2}\big{/}\|v(x^{*})\|^{4}\leq\|v(x^{*})\|^{2}\,\|v(x)\|^{2}\big{/}\|v(x^{*})\|^{4}=\|v(x)\|^{2}/\|v(x^{*})\|^{2}\,\leq 1 for all $x\in{\cal X}$. If x\in f_{\theta_{n}}^{-1}\bigl{(}\overline{V}(\delta)\bigr{)} then  $\|v(x)\|^{2}/\|v(x^{*})\|^{2}\leq\delta^{2}/\kappa$. Hence, partitioning ${\rm supp}(\xi_{n})$ into {\rm supp}(\xi_{n})\cap f_{\theta_{n}}^{-1}\bigl{(}\overline{V}(\delta)\bigr{)} and {\rm supp}(\xi_{n})\setminus f_{\theta_{n}}^{-1}\bigl{(}\overline{V}(\delta)\bigr{)}, one gets

[TABLE]

and together with (2.14) the result follows. $\Box$

We introduce a technical condition (T) which is weaker than (GLM). It is motivated by the result of Lemma 2.5 below.

**Condition (T)

**For each $\delta>0$ there exist an integer $m_{0}(\delta)\geq n_{\rm\scriptsize st}$ and a $\delta^{\prime}>0$ such that for all $k,\ell\geq m_{0}(\delta)$ and all linear subspaces $V\subseteq\mathbb{R}^{p}$ one has  f_{\theta_{k}}^{-1}\bigl{(}\overline{V}(\delta^{\prime})\bigr{)}\subseteq f_{\theta_{\ell}}^{-1}\bigl{(}\overline{V}(\delta)\bigr{)}. $\Box$

Lemma 2.5

*(i) Condition (GLM) implies condition (T).

(ii) If  $\lim_{n\to\infty}\theta_{n}=\overline{\theta}$  for some $\overline{\theta}\in\Theta$ then condition (T) holds.*

Proof.  Ad (i). Assume (GLM). Denote

[TABLE]

By compactness and continuity the infimum and the supremum are attained, and hence $0<\psi_{\rm\scriptsize min}\leq\psi_{\rm\scriptsize max}<\infty$. For a given $\delta>0$ choose $m_{0}(\delta)=n_{\rm\scriptsize st}$ and $\delta^{\prime}=\delta\,\psi_{\rm\scriptsize min}/\psi_{\rm\scriptsize max}$. Let $k,\ell\geq n_{\rm\scriptsize st}$ and a linear subspace $V\subseteq\mathbb{R}^{p}$ be given. For any $\theta\in\Theta$ and any $\varepsilon>0$ one has f_{\theta}^{-1}\bigl{(}\overline{V}(\varepsilon)\bigr{)}=\bigl{\{}x\in{\cal X}\,:\,{\rm dist}(\psi(x,\theta)\,f(x),V)\leq\varepsilon\bigr{\}}, and  {\rm dist}\bigl{(}\psi(x,\theta)\,f(x),V\bigr{)}=\psi(x,\theta)\,{\rm dist}(f(x),V), hence

[TABLE]

For $\theta=\theta_{k}$ and $\varepsilon=\delta^{\prime}$ (2.16) yields, observing $\delta^{\prime}/\psi(x,\theta_{k})\leq\delta^{\prime}/\psi_{\rm\scriptsize min}=\delta/\psi_{\rm\scriptsize max}$,

[TABLE]

For $\theta=\theta_{\ell}$ and $\varepsilon=\delta$ (2.16) yields, observing $\delta/\psi(x,\theta_{\ell})\geq\delta/\psi_{\rm\scriptsize max}$,

[TABLE]

From (2.17) and (2.18) the inclusion f_{\theta_{k}}^{-1}\bigl{(}\overline{V}(\delta^{\prime})\bigr{)}\subseteq f_{\theta_{\ell}}^{-1}\bigl{(}\overline{V}(\delta)\bigr{)} follows.

Ad (ii). Assume that $\lim_{n\to\infty}\theta_{n}=\overline{\theta}$ for some $\overline{\theta}\in\Theta$. By compactness of ${\cal X}\times\Theta$ and continuity (hence uniform continuity) of the function $(x,\theta)\longmapsto f_{\theta}(x)$ the sequence of functions $f_{\theta_{n}}$, $n\geq n_{\rm\scriptsize st}$, converges to $f_{\overline{\theta}}$ uniformly on ${\cal X}$. So, for any given $\delta>0$ there is an $m_{0}(\delta)\geq n_{\rm\scriptsize st}$ such that

[TABLE]

Choose $\delta^{\prime}=\delta/2$. Let $k,\ell\geq m_{0}(\delta)$ and a linear subspace $V\subseteq\mathbb{R}^{p}$ be given. Using the well-known inequality

[TABLE]

one gets from (2.19) that

[TABLE]

From (2.20), using $\delta^{\prime}=\delta/2$, one gets f_{\theta_{k}}^{-1}\bigl{(}\overline{V}(\delta^{\prime})\bigr{)}\subseteq f_{\theta_{\ell}}^{-1}\bigl{(}\overline{V}(\delta)\bigr{)}. $\Box$

Theorem 2.6

Assume condition (T). Then there exist $n_{0}\geq n_{\rm\scriptsize st}$, $\varepsilon>0$, and $\alpha\in(\,0\,,\,1\,)$ such that for all $n\geq n_{0}$ and all $(p-1)$-dimensional linear subspaces $V_{p-1}$ of $\mathbb{R}^{p}$ one has  \xi_{n}\Bigl{(}f_{\theta_{n}}^{-1}\bigl{(}\overline{V}_{p-1}(\varepsilon)\bigr{)}\Bigr{)}\leq\alpha.

Proof.  Firstly, consider the (nearly) trivial case $p=1$. The only [math]-dimensional linear subspace of $\mathbb{R}^{1}$ is $\{0\}$, hence $\overline{V}_{0}(\varepsilon)=[-\varepsilon\,,\,\varepsilon]$ for any $\varepsilon>0$. From (1.4) \bigl{(}f_{\theta_{n}}(x_{n+1})\bigr{)}^{2}=\max_{x\in{\cal X}}\bigl{(}f_{\theta_{n}}(x)\bigr{)}^{2} and by (2.8) \bigl{(}f_{\theta_{n}}(x_{n+1})\bigr{)}^{2}\geq\kappa for all $n\geq n_{\rm\scriptsize st}$. Choose a $\delta\in(\,0\,,\,\sqrt{\kappa}\,)$ and choose $m_{0}(\delta)\geq n_{\rm\scriptsize st}$ and $\delta^{\prime}>0$ according to condition (T). Then x_{n+1}\not\in f_{\theta_{n}}^{-1}\bigl{(}[\,-\delta\,,\,\delta\,]\bigr{)} for all $n\geq n_{\rm\scriptsize st}$, hence

[TABLE]

By (T), for all $n\geq m_{0}(\delta)$ the set f_{\theta_{n}}^{-1}\bigl{(}[\,-\delta^{\prime}\,,\,\delta^{\prime}\,]\bigr{)} is a subset of the intersection from (2.21) and hence x_{i}\not\in f_{\theta_{n}}^{-1}\bigl{(}[\,-\delta^{\prime}\,,\,\delta^{\prime}\,]\bigr{)} for all $i\geq m_{0}(\delta)+1$ and all $n\geq m_{0}(\delta)$. It follows that

[TABLE]

So, choosing $n_{0}=2m_{0}(\delta)$, $\varepsilon=\delta^{\prime}$, and $\alpha=1/2$, the statement of the theorem holds in case $p=1$. In what follows we assume $p\geq 2$. We will prove by induction the following statement ${\rm S}(r)$ for all $r=0,1,\ldots,p-1$.

• ${\rm S}(r)$

There exist $\widetilde{n}_{r}\geq n_{\rm\scriptsize st}$, $\varepsilon_{r}>0$, and $\alpha_{r}\in(\,0\,,\,1\,)$ such that  \xi_{n}\Bigl{(}f_{\theta_{n}}^{-1}\bigl{(}\overline{V}_{r}(\varepsilon_{r})\bigr{)}\Bigr{)}\leq\alpha_{r} for all $n\geq\widetilde{n}_{r}$ and all $r$-dimensional linear subspaces $V_{r}$ of $\mathbb{R}^{p}$.

Then the result will follow by taking $n_{0}=\widetilde{n}_{p-1}$, $\varepsilon=\varepsilon_{p-1}$, and $\alpha=\alpha_{p-1}$.

$r=0$.  The only [math]-dimensional linear subspace of $\mathbb{R}^{p}$ is the nullspace $V_{0}=\{0\}$, and for any $\varepsilon>0$ one has $\overline{V}_{0}(\varepsilon)=\{a\in\mathbb{R}^{p}\,:\,\|a\|\leq\varepsilon\}$, the closed ball centered at zero with radius $\varepsilon$. Choose any \eta\in\bigl{(}\,0\,,\,1-\frac{1}{\sqrt{p}}\,\bigr{)} and let $\delta:=\eta\kappa/(2\gamma)$. Choose $m_{0}(\delta)\geq n_{\rm\scriptsize st}$ and $\delta^{\prime}>0$ according to condition (T), and define

[TABLE]

Clearly, if x,z\in f_{\theta_{\ell}}^{-1}\bigl{(}\overline{V}_{0}(\delta)\bigr{)}, i.e., $\|f_{\theta_{\ell}}(x)\|\leq\delta$ and $\|f_{\theta_{\ell}}(z)\|\leq\delta$, then $\|f_{\theta_{\ell}}(x)-f_{\theta_{\ell}}(z)\|\leq 2\delta=\eta\kappa/\gamma$. So the subset $S$ has the property that if $n\geq m_{0}(\delta)$ and $x,z\in S$ then $\|f_{\theta_{n}}(x)-f_{\theta_{n}}(z)\|\leq\eta\kappa/\gamma$. By Lemma 2.3, if $n\geq m_{0}(\delta)$ and \xi_{n}(S)>1\big{/}\bigl{(}(1-\eta)^{2}p\bigr{)} then $x_{n+1}\not\in S$. Choose an $\alpha_{0}$ with 1\big{/}\bigl{(}(1-\eta)^{2}p\bigr{)}<\alpha_{0}<1. The sequence $\beta_{n}=\xi_{n}(S)$, $n\geq m_{0}(\delta)$, along with \beta=1\big{/}\bigl{(}(1-\eta)^{2}p\bigr{)} and $\widetilde{\beta}=\alpha_{0}$, satisfy the assumptions of Lemma 2.1, and hence by that lemma $\xi_{n}(S)\leq\widetilde{\beta}=\alpha_{0}$ for all n\geq m_{1}=m_{1}\bigl{(}\beta,\widetilde{\beta},m_{0}(\delta)\bigr{)}. By (T), f_{\theta_{n}}^{-1}\bigl{(}\overline{V}_{0}(\delta^{\prime})\bigr{)}\subseteq S for all $n\geq m_{0}(\delta)$ and hence \xi_{n}\Bigl{(}f_{\theta_{n}}^{-1}\bigl{(}\overline{V}_{0}(\delta^{\prime})\bigr{)}\Bigr{)}\leq\alpha_{0} for all $n\geq m_{1}$. So statement ${\rm S}(0)$ holds with $\widetilde{n}_{0}=m_{1}$, $\varepsilon=\delta^{\prime}$, and $\alpha_{0}$ as already introduced.

Induction step.  Suppose that for some $r\in\{1,\ldots,p-1\}$ statement ${\rm S}(r-1)$ is true, and let $\widetilde{n}_{r-1}$, $\varepsilon_{r-1}$, and $\alpha_{r-1}$ be chosen as in statement ${\rm S}(r-1)$. Since every linear subspace $V_{t}\subseteq\mathbb{R}^{p}$ of dimension $t\leq r-1$ can be enlarged to an $(r-1)$-dimensional linear subspace $V_{r-1}\subseteq\mathbb{R}^{p}$, where $V_{t}\subseteq V_{r-1}$ and hence $\overline{V}_{t}(\varepsilon_{r-1})\subseteq\overline{V}_{r-1}(\varepsilon_{r-1})$, the assumed statement ${\rm S}(r-1)$ implies the following.

[TABLE]

The rest of the proof of the induction step is lengthy; it is structured into three steps.

Step 1.  We introduce some sets and constants.

[TABLE]

Obviously, ${\cal A}_{r}$ and ${\cal A}_{r}^{*}$ are compact sets of $p\times r$ matrices. It is not quite obvious that ${\cal A}_{r}^{*}$ is nonempty which can be seen as follows. (2.22) implies in particular that, choosing any $n\geq\widetilde{n}_{r-1}$, the set {\cal X}\setminus f_{\theta_{n}}^{-1}\bigl{(}\overline{V}_{0}(\varepsilon_{r-1})\bigr{)} is nonempty, i.e., there is a $z\in{\cal X}$ such that $\|f_{\theta_{n}}(z)\|>\varepsilon_{r-1}$. By $\|f_{\theta_{n}}(z)\|\leq\gamma$ one has $\varepsilon_{r-1}<\gamma$. Choosing pairwise orthogonal vectors $a_{1},\ldots,a_{r}\in\mathbb{R}^{p}$ with $\|a_{j}\|=\gamma$, $1\leq j\leq r$, one gets a matrix $A=[a_{1},\ldots,a_{r}]\in{\cal A}_{r}$ with $\det(A^{\sf\scriptsize T}A)=\gamma^{2r}>\varepsilon_{r-1}^{2r}$, hence $A\in{\cal A}_{r}^{*}$. Next, denote by $\|w\|_{1}\,=\,\sum_{j=1}^{r}|w_{j}|$ the $\ell^{1}$-norm of a vector $w=(w_{1},\ldots,w_{r})^{\sf\scriptsize T}\in\mathbb{R}^{r}$ and define

[TABLE]

Choosing any $A=[a_{1},\ldots,a_{r}]\in{\cal A}_{r}^{*}$ and $b=a_{1}$ gives $(A^{\sf\scriptsize T}A)^{-1}A^{\sf\scriptsize T}b=e_{1}=(1,0,\ldots,0)^{\sf\scriptsize T}$, and $c_{r}\geq 1$ follows. Together with compactness and continuity one has $1\leq c_{r}<\infty$. Again by compactness and continuity one can choose a positive integer $K_{r}$ and nonempty subsets $R_{1},\ldots,R_{K_{r}}\subseteq{\cal X}$ such that

[TABLE]

[TABLE]

Note that \alpha_{r-1}\,<\,(K_{r}c_{r}^{2}+\alpha_{r-1})\big{/}(K_{r}c_{r}^{2}+1), hence $\alpha_{r-1}<\overline{\alpha}_{r}$. Finally, choose a $\delta>0$ which satisfies the following three conditions,

[TABLE]

In fact, such a $\delta$ exists since, firstly, both sides of the inequality (2.29) are continuous functions of a real variable $\delta$ and the (strict) inequality (2.29) holds for $\delta=0$ by (2.27). Secondly, (2.30) is achieved by the uniform continuity of the function $A\longmapsto\det(A^{\sf\scriptsize T}A)$ on the compact set ${\cal A}_{r}$ from (2.23).

Step 2.  With $\delta$ and $\overline{\alpha}_{r}$ from Step 1 we show the following:

• *If $V_{r}\subseteq\mathbb{R}^{p}$ is an $r$-dimensional linear subspace and $n\geq\widetilde{n}_{r-1}$ such that

\xi_{n}\Bigl{(}f_{\theta_{n}}^{-1}\bigl{(}\overline{V}_{r}(\delta)\bigr{)}\Bigr{)}\,>\,\overline{\alpha}_{r},  then  x_{n+1}\not\in f_{\theta_{n}}^{-1}\bigl{(}\overline{V}_{r}(\delta)\bigr{)}.*

Let an $r$-dimensional linear subspace $V_{r}\subseteq\mathbb{R}^{p}$ and an $n\geq\widetilde{n}_{r-1}$ be given such that

[TABLE]

By property (2.28), \delta^{2}/\kappa<\kappa\big{/}\bigl{(}(c_{r}+1)^{2}\gamma^{2}\bigr{)}\leq\kappa/\gamma^{2}\leq 1, where the last inequality is obvious by the definitions of $\gamma$ and $\kappa$ in (2.7) and (2.8). So $\delta<\sqrt{\kappa}$ and by Lemma 2.4 and (2.31)

[TABLE]

Next, we construct a particular basis $b_{1},\ldots,b_{r}$ of the linear subspace $V_{r}$. From (2.22) and (2.31) it follows that for all linear subspaces $V_{t}\subseteq\mathbb{R}^{p}$ of dimension $t\leq r-1$ one has

[TABLE]

Note that by (2.26), in particular, the sets $R_{1},\ldots,R_{K_{r}}$ cover ${\cal X}$. Thus (2.33) implies that to any linear subspace $V\subseteq\mathbb{R}^{p}$ of dimension at most $r-1$ one can find some index $k\in\{1,\ldots,K_{r}\}$ such that \xi_{n}\Bigl{(}f_{\theta_{n}}^{-1}\bigl{(}\overline{V}_{r}(\delta)\setminus\overline{V}(\varepsilon_{r-1})\bigr{)}\cap R_{k})\Bigr{)}\,>\,(\overline{\alpha}_{r}-\alpha_{r-1})/K_{r}. Using this, one obtains inductively $r$ subsets $S_{1},\ldots,S_{r}$ of ${\cal X}$ such that for all $j=1,\ldots,r$,

[TABLE]

with particular linear subspaces $W_{0},\ldots,W_{r-1}$ given by

[TABLE]

where $\overline{f}_{\theta_{n}}(S_{i},\xi_{n})$ denotes the average of $f_{\theta_{n}}(x)$ over $x\in S_{i}$ w.r.t. $\xi_{n}$ analogously to (2.13). For each $j=1,\ldots,r$ by (2.34), firstly, $f_{\theta_{n}}(x)\in\overline{V}_{r}(\delta)$ for all $x\in S_{j}$ and hence also for the mean $\overline{f}_{\theta_{n}}(S_{j},\xi_{n})\in\overline{V}_{r}(\delta)$ since the set $\overline{V}_{r}(\delta)$ is convex. Secondly, $f_{\theta_{n}}(x)\not\in\overline{W}_{j-1}(\varepsilon_{r-1})$ for all $x\in S_{j}$, i.e., ${\rm dist}(f_{\theta_{n}}(x),W_{j-1})>\varepsilon_{r-1}$ for all $x\in S_{j}$. Thirdly, $S_{j}\subseteq R_{k_{j}}$, hence $\|f_{\theta_{n}}(x)-f_{\theta_{n}}(z)\|\leq\varepsilon_{r-1}/2$ for all $x,z\in S_{j}$ which implies $\|f_{\theta_{n}}(x)-\overline{f}_{\theta_{n}}(S_{j},\xi_{n})\|\leq\varepsilon_{r-1}/2$ for all $x\in S_{j}$. Using the inequality \big{|}{\rm dist}(f_{\theta_{n}}(x),W_{j-1})-{\rm dist}\bigl{(}\overline{f}_{\theta_{n}}(S_{j},\xi_{n}),W_{j-1}\bigr{)}\big{|}\leq\|f_{\theta_{n}}(x)-\overline{f}_{\theta_{n}}(S_{j},\xi_{n})\| one gets, choosing any $x\in S_{j}$,

[TABLE]

By (2.36) together with (M4) of the appendix the $p\times r$ matrix F:=\bigl{[}\overline{f}_{\theta_{n}}(S_{1},\xi_{n}),\ldots,\overline{f}_{\theta_{n}}(S_{r},\xi_{n})\bigr{]} satisfies

[TABLE]

For each $j=1,\ldots,r$, by $\overline{f}_{\theta_{n}}(S_{j},\xi_{n})\in\overline{V}_{r}(\delta)$,

[TABLE]

Consider the $p\times r$ matrix B:=\bigl{[}b_{1},\ldots,b_{r}\bigr{]}. Since $\|b_{j}\|\leq\|\overline{f}_{\theta_{n}}(S_{j},\xi_{n})\|\leq\gamma$ one has $F,B\in{\cal A}_{r}$. By (2.38) $\|\overline{f}_{\theta_{n}}(S_{j},\xi_{n})-b_{j}\|\leq\delta$, $1\leq j\leq r$, and hence, using property (2.30) of $\delta$ and (2.37),

[TABLE]

In particular, $B\in{\cal A}_{r}^{*}$ and the vectors $b_{1},\ldots,b_{r}$ are linearly independent and form thus a basis of the linear subspace $V_{r}$. Now suppose, contrary to the assertion of Step 2, that x_{n+1}\in f_{\theta_{n}}^{-1}\bigl{(}\overline{V}_{r}(\delta)\bigr{)}. Then

[TABLE]

Since $b_{1},\ldots,b_{r}$ constitute a basis of $V_{r}$ and B=\bigl{[}b_{1},\ldots,b_{r}\bigr{]}, one has  $v\,=\,Bw$ for some $w=(w_{1},\ldots,w_{r})^{\sf\scriptsize T}\in\mathbb{R}^{r}$. In fact, $w$ is uniquely determined by $w=(B^{\sf\scriptsize T}B)^{-1}B^{\sf\scriptsize T}v$. Since $\|v\|\leq\|f_{\theta_{n}}(x_{n+1})\|\leq\gamma$ and $B\in{\cal A}_{r}^{*}$ one has, according to the definition of $c_{r}$ in (2.25), that $\|w\|_{1}\leq c_{r}$. Together with (2.38),

[TABLE]

[TABLE]

Define  $a\,:=\,\sum_{j=1}^{r}w_{j}\overline{f}_{\theta_{n}}(S_{j},\xi_{n})$. Hence

[TABLE]

Let $\eta:=(c_{r}+1)\,\delta\,\gamma/\kappa$. Then $\eta\in(\,0\,,\,1\,)$ by property (2.28) of $\delta$, and by (2.39) $\|f_{\theta_{n}}(x_{n+1})-a\|\leq\eta\kappa/\gamma$. So, by Lemma 2.2,

[TABLE]

Observing that b\mapsto\bigl{(}b^{\sf\scriptsize T}M^{-1}(\xi_{n},\theta_{n})\,b\bigr{)}^{1/2}, $b\in\mathbb{R}^{p}$, is a norm on $\mathbb{R}^{p}$ and using the definition of the vector $a$,

[TABLE]

For each $j=1,\ldots,r$, one gets by (M1) and (M2) of the appendix, where the sums below are taken over $x\in S_{j}\cap\,{\rm supp}(\xi_{n})$,

[TABLE]

where the last inequality is due to (2.35). Hence by (2.41) and by $\|w\|_{1}\leq c_{r}$,

[TABLE]

and together with (2.40) one gets

[TABLE]

Observing that the r.h.s. of (2.42) equals the reciprocal of the l.h.s. of (2.29), it follows from (2.29) that

[TABLE]

and hence by (2.42)

[TABLE]

which is a contradiction to (2.32) derived above. So our supposition that x_{n+1}\in f_{\theta_{n}}^{-1}\bigl{(}\overline{V}_{r}(\delta)\bigr{)} was wrong. Hence the result of Step 2 follows.

Step 3.  For $\delta$ from the previous Steps 1 and 2, let $m_{0}(\delta)$ and $\delta^{\prime}>0$ be chosen according to (T), where we may assume that $m_{0}(\delta)\geq\widetilde{n}_{r-1}$. Recall that $0<\overline{\alpha}_{r}<1$ according to (2.27). Choose an $\alpha_{r}$ such that $\overline{\alpha}_{r}<\alpha_{r}<1$. Let $V_{r}$ be any $r$-dimensional linear subspace of $\mathbb{R}^{p}$. Consider the set

[TABLE]

By the result of Step 2 and by  S\subseteq f_{\theta_{n}}^{-1}\bigl{(}\overline{V}_{r}(\delta)\bigr{)} for all $n\geq m_{0}(\delta)$, we have:

[TABLE]

So the sequence

[TABLE]

satisfies the assumptions of Lemma 2.1, and hence by that lemma $\beta_{n}\leq\alpha_{r}$ for all n\geq m_{1}=m_{1}\bigl{(}\beta,\widetilde{\beta},m_{0}(\delta)\bigr{)}. Since $m_{1}$ does not depend on the particular choice of $V_{r}$ we have thus obtained that for all linear subspaces $V_{r}\subseteq\mathbb{R}^{p}$ of dimension $r$ one has

[TABLE]

According to (T) we have $\delta^{\prime}>0$ such that f_{\theta_{n}}^{-1}\bigl{(}\overline{V}_{r}(\delta^{\prime})\bigr{)}\subseteq\bigcap_{\ell\geq m_{0}(\delta)}f_{\theta_{\ell}}^{-1}\bigl{(}\overline{V}_{r}(\delta)\bigr{)} for all $n\geq m_{1}$ and all linear subspaces $V_{r}$ of dimension $r$. Hence \xi_{n}\Bigl{(}f_{\theta_{n}}^{-1}\bigl{(}\overline{V}_{r}(\delta^{\prime})\bigr{)}\Bigr{)}\leq\alpha_{r} for all $n\geq m_{1}$ and all linear subspaces $V_{r}$ of dimension $r$, which is statement ${\rm S}(r)$ with $\widetilde{n}_{r}:=m_{1}$, $\varepsilon_{r}:=\delta^{\prime}$, and $\alpha_{r}$ as obtained. So the induction step has been completed. $\Box$

Corollary 2.7

*(i) If (T) is satisfied then the asymptotic nonsingularity (2.1) holds.

(ii) If (GLM) is satisfied then the stronger asymptotic nonsingularity (2.2) holds.*

Proof.  Using a well-known representation of the smallest eigenvalue of a symmetric matrix we can write

[TABLE]

For any $c\in\mathbb{R}^{p}$, $\|c\|=1$, we denote by $V_{p-1,c}$ the $(p-1)$-dimensional subspace of $\mathbb{R}^{p}$ given by V_{p-1,c}\,=\,\bigl{\{}a\in\mathbb{R}^{p}\,:\,c^{\sf\scriptsize T}a=0\bigr{\}}. Assume (T). Let $n_{0}\geq n_{\rm\scriptsize st}$, $\varepsilon>0$, and $\alpha\in(\,0\,,\,1\,)$ be chosen according to Theorem 2.6. Then by the theorem, observing that  f^{-1}_{\theta_{n}}\bigl{(}\overline{V}_{p-1,c}(\varepsilon)\bigr{)}\,=\,\bigl{\{}x\in{\cal X}\,:\,|c^{\sf\scriptsize T}f_{\theta_{n}}(x)|\leq\varepsilon\bigr{\}}, we have for all $n\geq n_{0}$ and all $c$ with $\|c\|=1$,

[TABLE]

Denote S_{n,c}\,=\,\bigl{\{}x\in{\cal X}\,:\,|c^{\sf\scriptsize T}f_{\theta_{n}}(x)|>\varepsilon\bigr{\}}. Then for all $n\geq n_{0}$ and all $c\in\mathbb{R}^{p}$ with $\|c\|=1$ one has $\xi_{n}(S_{n,c})\geq 1-\alpha$ and hence

[TABLE]

and together with (2.43), \lambda_{\rm\scriptsize min}\bigl{(}M(\xi_{n},\theta_{n})\bigr{)}\geq\widetilde{\lambda}_{0} for all $n\geq n_{0}$. So, in the case $n_{0}>n_{\rm\scriptsize st}$, a positive real constant is given by \lambda_{0}:=\min\{\widetilde{\lambda}_{0},\lambda_{\rm\scriptsize min}\bigl{(}M(\xi_{n},\theta_{n})\bigr{)}:\,n_{\rm\scriptsize st}\leq n<n_{0}\}. In the case $n_{0}=n_{\rm\scriptsize st}$ choose $\lambda_{0}:=\widetilde{\lambda}_{0}$. In any case, with that constant $\lambda_{0}>0$ the asymptotic nonsingularity (2.1) holds. Now assume (GLM). By Lemma 2.4 (T) is satisfied as well and hence, as already proved, the asymptotic nonsingularity (2.1) holds with some $\lambda_{0}>0$. For all $x\in{\cal X}$ and all $\theta\in\Theta$ one has $f_{\theta}(x)\,f_{\theta}^{\sf\scriptsize T}(x)=\psi^{2}(x,\theta)\,f(x)\,f^{\sf\scriptsize T}(x)$. Consider the positive real numbers $\psi_{\rm\scriptsize min}$ and $\psi_{\rm\scriptsize max}$ from (2.15). Then, for all $x\in{\cal X}$ and $\theta,\theta^{\prime}\in\Theta$ trivially $(\psi_{\rm\scriptsize min}/\psi_{\rm\scriptsize max})^{2}\psi^{2}(x,\theta^{\prime})\leq\psi^{2}(x,\theta)$. Hence for any design $\xi$ one has $(\psi_{\rm\scriptsize min}/\psi_{\rm\scriptsize max})^{2}M(\xi,\theta^{\prime})\leq M(\xi,\theta)$ for all $\theta,\theta^{\prime}\in\Theta$. In particular, one has $(\psi_{\rm\scriptsize min}/\psi_{\rm\scriptsize max})^{2}M(\xi_{n},\theta_{n})\leq M(\xi_{n},\theta)$ for all $n\geq n_{\rm\scriptsize st}$ and $\theta\in\Theta$. It follows that

[TABLE]

So the stronger asymptotic nonsingularity (2.2) holds with $(\psi_{\rm\scriptsize min}/\psi_{\rm\scriptsize max})^{2}\lambda_{0}$ instead of $\lambda_{0}$. $\Box$

As a further consequence from Theorem 2.6 and Corollory 2.7 we can derive a convergence result as in Pronzato [11], Lemma 2 and Theorem 2, and Freise [6], Lemma 18. If the sequence of parameter points $\theta_{n}$ converges to some parameter point $\overline{\theta}\in\Theta$ then the design sequence $\xi_{n}$ is asymptotically locally D-optimal at $\overline{\theta}$, in the sense that the sequence of information matrices $M(\xi_{n},\theta_{n})$ converges to the information matrix $M(\xi^{*}_{\overline{\theta}},\overline{\theta})$ of a locally D-optimal design $\xi^{*}_{\overline{\theta}}$ at $\overline{\theta}$. For later reference (see Section 3), the next theorem states the convergence of the information matrices $M(\xi_{n},\theta_{n}^{\prime})$ to $M(\xi^{*}_{\overline{\theta}},\overline{\theta})$ for any sequence $\theta_{n}^{\prime}\in\Theta$ converging to $\overline{\theta}$, provided that the sequence $\theta_{n}$ employed by the algorithm converges to $\overline{\theta}$. Of course, in the linear model case, $f_{\theta}=f$ identical for all $\theta\in\Theta$, we retrieve the classical result of Wynn [16], Theorem 1.

Theorem 2.8

*If  $\lim_{n\to\infty}\theta_{n}=\overline{\theta}$  for some $\overline{\theta}\in\Theta$ then for every sequence $\theta_{n}^{\prime}\in\Theta$, $n\geq n_{\rm\scriptsize st}$, such that

$\lim_{n\to\infty}\theta_{n}^{\prime}=\overline{\theta}$ one has*

[TABLE]

where $\xi^{*}_{\overline{\theta}}$ denotes a locally D-optimal design at $\overline{\theta}$, i.e., $\xi^{*}_{\overline{\theta}}$ maximizes \det\bigl{(}M(\xi,\overline{\theta})\bigr{)} over the set of all designs $\xi$.

Proof.  The matrix-valued function  $(x,\theta)\longmapsto f_{\theta}(x)\,f_{\theta}^{\sf\scriptsize T}(x)$  is uniformly continuous on its compact domain ${\cal X}\times\Theta$. So, for any sequence $\theta_{n}^{\prime}\in\Theta$ converging to $\overline{\theta}$, observing that $M(\xi_{n},\theta)=\frac{1}{n}\sum_{i=1}^{n}f_{\theta}(x_{i})\,f_{\theta}^{\sf\scriptsize T}(x_{i})$ for all $n\geq n_{\rm\scriptsize st}$ and $\theta\in\Theta$,

[TABLE]

Hence

[TABLE]

and, in particular,

[TABLE]

Consider $\gamma$ from (2.7). For any design $\xi$ and any $\theta\in\Theta$ we have

[TABLE]

By Lemma 2.4 and Corollary 2.7 there is a $\lambda_{0}>0$ satisfying (2.1). Let ${\cal A}$ be the set of all nonnegative definite $p\times p$ martices $A$ such that $\lambda_{\rm\scriptsize min}(A)\geq\lambda_{0}/2$ and ${\rm tr}(A)\leq\gamma^{2}$. Clearly, ${\cal A}$ is compact and $M(\xi_{n},\theta_{n})\in{\cal A}$ for all $n\geq n_{\rm\scriptsize st}$, and by (2.45) there is an $m_{0}\geq n_{\rm\scriptsize st}$ such that $M(\xi_{n},\overline{\theta})\in{\cal A}$ for all $n\geq m_{0}$. Define a real-valued function $H$ on ${\cal X}\times\Theta\times{\cal A}$ by

[TABLE]

which is continuous and hence uniformly continuous on its compact domain ${\cal X}\times\Theta\times{\cal A}$. So, together with (2.45),

[TABLE]

In what follows let an $\varepsilon\in(\,0\,,\,1\,)$ be given. By (2.46) and by the definition of the function $H$, there is an $n_{0}\geq m_{0}$ such that

[TABLE]

This yields, in particular,

(i) $\forall n\geq n_{0}$ :  $\displaystyle f^{\sf\scriptsize T}_{\overline{\theta}}(x_{n+1})\,M^{-1}(\xi_{n},\overline{\theta})\,f_{\overline{\theta}}(x_{n+1})\,\geq\,\max_{x\in{\cal X}}f^{\sf\scriptsize T}_{\overline{\theta}}(x)\,M^{-1}(\xi_{n},\overline{\theta})\,f_{\overline{\theta}}(x)\,-\,\varepsilon\,\geq\,p-\varepsilon$,

since for all $n\geq n_{0}$, denoting  $x_{n}^{*}=\arg\max_{x\in{\cal X}}f_{\overline{\theta}}^{\sf\scriptsize T}(x)\,M^{-1}(\xi_{n},\overline{\theta})\,f_{\overline{\theta}}(x)$,

[TABLE]

The second inequality in (i) is well-known from the Kiefer-Wolfowitz Equivalence Theorem. The rest of the proof employs the arguments of Pronzato [11] in the proof of Lemma 3 of that paper. For convenience we report here the main steps labelled below by (ii) - (v).

(ii) One can choose $n_{1}\geq n_{0}$ such that for all $n\geq n_{1}$

[TABLE]

To see this we note that  $(n+1)M(\xi_{n+1},\overline{\theta})=nM(\xi_{n},\overline{\theta})+f_{\overline{\theta}}(x_{n+1})\,f^{\sf\scriptsize T}_{\overline{\theta}}(x_{n+1})$ and by a well-known formula of determinants,

[TABLE]

By (i) for $n\geq n_{0}$ the expression (2.47) is greater than or equal to

[TABLE]

where we have used that $(1+1/n)^{p}=1+(p+c_{n})/n$ with $c_{n}>0$, $c_{n}\to 0$ as $n\to\infty$. Choose $n_{1}\geq n_{0}$ such that $c_{n}\leq(p-\varepsilon)\varepsilon$ for all $n\geq n_{1}$. Then for all $n\geq n_{1}$,

[TABLE]

(iii) One can choose $n_{2}\geq n_{1}$ such that for all $n\geq n_{2}$

[TABLE]

This follows from $(1+1/n)^{p}=1+(p+c_{n})/n$ and by choosing $n_{2}\geq n_{1}$ such that c_{n}\leq\varepsilon\bigl{(}1-\frac{n+p+\varepsilon}{2n}\bigr{)} for all $n\geq n_{2}$.

(iv) Denote \Psi^{*}:=\log\det\bigl{(}M(\xi^{*}_{\overline{\theta}},\overline{\theta})\bigr{)}. If $n\geq n_{2}$ and  \log\det\bigl{(}M(\xi_{n},\overline{\theta})\bigr{)}\,\leq\ \Psi^{*}-2\varepsilon  then

[TABLE]

This can be seen as follows. By the gradient inequality for the concave criterion $\log\det(\,\cdot\,)$,

[TABLE]

where the last inequality comes from (i). Hence it follows that $f_{\overline{\theta}}^{\sf\scriptsize T}(x_{n+1})M^{-1}(\xi_{n},\overline{\theta})\,f_{\overline{\theta}}(x_{n+1})\geq p+\varepsilon$ and together with (2.47) one gets

[TABLE]

where the last inequality comes from (iii).

(v) One can choose $n_{3}\geq n_{2}$ such that for all $n\geq n_{3}$

[TABLE]

To see this, note that by (iv) there is some $n_{3}\geq n_{2}$ such that  \log\det\bigl{(}M(\xi_{n_{3}},\overline{\theta})\bigr{)}>\Psi^{*}-2\varepsilon, since otherwise (iv) would yield that \log\det\bigl{(}M(\xi_{n},\overline{\theta})\bigr{)}\longrightarrow\infty as $n\to\infty$, which is a contradiction. By (ii) and (iv), the sequence  a_{n}:=\log\det\bigl{(}M(\xi_{n},\overline{\theta})\bigr{)}, $n\geq n_{3}$, has the following properties.

[TABLE]

Thus, obviously, $a_{n}>\Psi^{*}-3\varepsilon$ for all $n\geq n_{3}$, which is (v).

From (v) we get

[TABLE]

Since $\varepsilon\in(\,0\,,\,1\,)$ was arbitrary we get  \liminf_{n\to\infty}\log\det\bigl{(}M(\xi_{n},\overline{\theta})\bigr{)}\,\geq\,\Psi^{*}  and hence  \lim_{n\to\infty}\log\det\bigl{(}M(\xi_{n},\overline{\theta})\bigr{)}\,=\,\Psi^{*}. This implies  $\lim_{n\to\infty}M(\xi_{n},\overline{\theta})\,=\,M(\xi^{*}_{\overline{\theta}},\overline{\theta})$, since by strict concavity of the criterion $\log\det(\,\cdot\,)$ the information matrix at $\overline{\theta}$ of a locally D-optimal design at $\overline{\theta}$ is unique. That is, denoting by $\Xi$ the set of all designs and {\cal M}_{\overline{\theta}}:=\bigl{\{}M(\xi,\overline{\theta})\,:\,\mbox{\xi\in\Xi }\bigr{\}}, the set of all information matrices of designs at $\overline{\theta}$, the information matrix $M^{*}=M(\xi^{*}_{\overline{\theta}},\overline{\theta})$ is the unique point in ${\cal M}_{\overline{\theta}}$ such that  $\log\det(M^{*})\,=\max_{M\in{\cal M}_{\overline{\theta}}}\log\det(M)$. So for any $\delta>0$ one has by compactness and continuity

[TABLE]

So, \lim_{n\to\infty}\log\det\bigl{(}M(\xi_{n},\overline{\theta}))\,=\log\det(M^{*}) implies $\lim_{n\to\infty}M(\xi_{n},\overline{\theta})\,=M^{*}$. If $\theta_{n}^{\prime}\in\Theta$ is any sequence converging to $\overline{\theta}$ then by (2.44) $\lim_{n\to\infty}M(\xi_{n},\theta_{n}^{\prime})\,=M^{*}$. $\Box$

3 Adaptive Wynn-algorithm in univariate GLM

Now we focus on the adaptive character of the algorithm. The sequence of parameter points $\theta_{n}$, $n\geq n_{\rm\scriptsize st}$, employed is given by parameter estimates based on the data available at the current stage $n$, which are the design points $x_{1},\ldots,x_{n}$ and the observed values $y_{1},\ldots,y_{n}$ of a univariate response variable. We assume a (nonlinear) regression model with expected univariate responses $\mu(x,\theta)$, where $x\in{\cal X}$ and $\theta\in\Theta$. The function $\mu\,:\,{\cal X}\times\Theta\longrightarrow\mathbb{R}$ is assumed to be continuous and, as in the previous sections, the experimental region ${\cal X}$ and the parameter space $\Theta$ are compact metric spaces. Again, for the algorithm we assume a family $f_{\theta}$, $\theta\in\Theta$, of $\mathbb{R}^{p}$-valued functions on ${\cal X}$ defining the information matrices of designs by (1.1) and having the properties that for each $\theta\in\Theta$ the image $f_{\theta}({\cal X})$ spans $\mathbb{R}^{p}$, and the function $(x,\theta)\longmapsto f_{\theta}(x)$ is continuous on ${\cal X}\times\Theta$. The adaptive Wynn-algorithm sequentially generates data $x_{1},y_{1},\ldots,x_{n},y_{n},\ldots$ where $y_{i}$ is the observed (univariate) response at the design point $x_{i}$ ($i=1,2,\ldots,n,\ldots$) and the employed sequence $\theta_{n}$, $n\geq n_{\rm\scriptsize st}$, is given by adaptive parameter estimates,  $\widehat{\theta}_{n}=\widehat{\theta}_{n}(x_{1},y_{1},\ldots,x_{n},y_{n})$, $n\geq n_{\rm\scriptsize st}$. In particular, the values $y_{i}$ of the response variable as well as the generated values $x_{i}$ of the design variable are random and hence they are modelled by random variables $Y_{i}$ and $X_{i}$. The sequential and adaptive character of the data is caught by the ‘adaptive regression model’ formulated and discussed in Subsection 3.1 below. For theoretical investigations on consistency or asymptotic distribution of estimators it will be convenient to distinguish between the true (but unknown) parameter point $\overline{\theta}$ and any possible parameter point $\theta\in\Theta$ to be considered. So throughout this section, $\overline{\theta}\in\Theta$ denotes the fixed true parameter point governing the random variables.

3.1 Adaptive regression model.

An appropriate model for the adaptive character of the sequences of random variables $X_{i}$ and $Y_{i}$, $i\in\mathbb{N}$, is provided by the following assumptions (A1) and (A2), cp. Lai [9], Sec. 1, or Chen, Hu, and Ying [4], Sec. 3. Note that all the random variables are defined on some probability space $(\Omega,{\cal F},\mathbb{P}_{\overline{\theta}})$, where $\Omega$ is a nonempty set, ${\cal F}$ is a sigma-field of subsets of $\Omega$, and $\mathbb{P}_{\overline{\theta}}$ is a probability measure on ${\cal F}$ corresponding to the true parameter point $\overline{\theta}$.

• (A1)

There is given a nondecreasing sequence of sub-sigma-fields of ${\cal F}$,  ${\cal F}_{0}\subseteq{\cal F}_{1}\subseteq\,\ldots\,\subseteq{\cal F}_{n}\subseteq,\ldots$ such that for each $i\in\mathbb{N}$ the random variable $X_{i}$ is ${\cal F}_{i-1}$-measurable and the random variable $Y_{i}$ is ${\cal F}_{i}$-measurable.

• (A2)

$Y_{i}\,=\,\mu(X_{i},\overline{\theta})\,+\,e_{i}$  with real-valued square integrable random errors $e_{i}$ such that

{\rm E}\bigl{(}e_{i}\,\big{|}\,{\cal F}_{i-1}\bigr{)}\,=0\ \mbox{\,a.s.}  for all $i\in\mathbb{N}$, and  \sup_{i\in\mathbb{N}}{\rm E}\bigl{(}e_{i}^{2}\,\big{|}\,{\cal F}_{i-1}\bigr{)}\,<\infty\ \ \mbox{a.s.}

As an illustration of the sub-sigma-fields ${\cal F}_{i}$, $i\in\mathbb{N}_{0}$, suppose that the starting design $\xi_{n_{\rm\tiny st}}$ of the algorithm was chosen deterministically, i.e., $X_{1},\ldots,X_{n_{\rm\tiny st}}$ are constants, and suppose further that for all $n\geq n_{\rm\scriptsize st}$ there is no ambiguity in chosing the maximizer $x_{n+1}$ in (1.4) given the values of $X_{1},Y_{1},\ldots,X_{n},Y_{n}$ and thus given the value of $\widehat{\theta}_{n}$. Then for all $n\geq n_{\rm\scriptsize st}$ the random variable $X_{n+1}$ is a function of $Y_{1},\ldots,Y_{n}$. So one can employ the particular sigma-fields  ${\cal F}_{n}=\sigma(Y_{1},\ldots,Y_{n})$ generated by the random variables $Y_{1},\ldots,Y_{n}$, for all $n\geq 1$, and ${\cal F}_{0}$ the minimal sigma-field in $\Omega$. We note that no further relation is assumed so far between the mean response function $\mu$ and the family of functions $f_{\theta}$, $\theta\in\Theta$, of the algorithm, whereas a particular relation will be employed in the next subsection.

The following lemma presents some auxiliary asymptotic results derived from martingale limit theorems. If $W_{n}$, $n\in\mathbb{N}$, is a sequence of $\mathbb{R}^{k}$-valued random variables and $W$ is an $\mathbb{R}^{k}$-valued random variable, the notation  $W_{n}\,\stackrel{{\scriptstyle\rm\scriptsize a.s.}}{{\longrightarrow}}\,\,W$  stands for almost sure convergence of the sequence $W_{n}$ to $W$ (as $n\to\infty$). For real-valued $W_{n}$ we will also use the notation $W_{n}\,\stackrel{{\scriptstyle\rm\scriptsize a.s.}}{{\longrightarrow}}\,\,\infty$   for indicating almost sure convergence (or ‘divergence’) to infinity.

Lemma 3.1

*Under(A1) and (A2) the following (a), (b), and (c) hold.

(a)  $\displaystyle\limsup_{n\to\infty}\,\frac{1}{n}\,\sum_{i=1}^{n}|e_{i}|\ <\,\infty\ \ \mbox{a.s.}$

(b)  Let $Z_{i}$, $i\in\mathbb{N}$, be a sequence of real-valued square integrable random variables such that $Z_{i}$ is ${\cal F}_{i-1}$-measurable for all $i\in\mathbb{N}$ and  $\sup_{i\in\mathbb{N}}|Z_{i}|\,<\infty\,\mbox{ a.s.}$ Then*

[TABLE]

(c)*  Let  $h\,:\,{\cal X}\times\Theta\longrightarrow\mathbb{R}$ be a continuous function. Then*

[TABLE]

**Proof.

(a)**  Denote  W_{i}\,:=\,|e_{i}|-{\rm E}\bigl{(}|e_{i}|\,\big{|}{\cal F}_{i-1}\bigr{)}, $i\in\mathbb{N}$. It is easily seen that the sequence of partial sums  $\sum_{i=1}^{n}W_{i}$, $n\in\mathbb{N}$, is a martingale w.r.t. ${\cal F}_{n}$, $n\in\mathbb{N}$. Since

[TABLE]

one has by (A2)  \sup_{i\in\mathbb{N}}{\rm E}\bigl{(}W_{i}^{2}\big{|}{\cal F}_{i-1}\bigr{)}\,<\infty\ \,\mbox{a.s.} and hence  \sum_{i=1}^{\infty}i^{-2}{\rm E}\bigl{(}W_{i}^{2}\big{|}{\cal F}_{i-1}\bigr{)}\,<\infty\ \,\mbox{a.s.} By Theorem 2.18 of Hall and Heyde [8],  $\frac{1}{n}\sum_{i=1}^{n}W_{i}\,\,\stackrel{{\scriptstyle\rm\scriptsize a.s.}}{{\longrightarrow}}\,0$, i.e.,

[TABLE]

By (A2) and Jensen’s inequality  \sup_{i\in\mathbb{N}}{\rm E}\bigl{(}|e_{i}|\,\big{|}{\cal F}_{i-1}\bigr{)}\,<\infty\ \,\mbox{a.s.}  from which one gets

$\limsup_{n\to\infty}\frac{1}{n}\sum_{i=1}^{n}|e_{i}|\,<\infty\ \,\mbox{a.s.}$

(b)  As it is easily seen, the sequence  $\sum_{i=1}^{n}Z_{i}e_{i}$, $n\in\mathbb{N}$, is a martingale w.r.t. ${\cal F}_{n}$, $n\in\mathbb{N}$. By assumption there are two real random variables $U_{1}$ and $U_{2}$ such that U_{1}\,=\,\sup_{i\in\mathbb{N}}{\rm E}\bigl{(}e_{i}^{2}\,\big{|}\,{\cal F}_{i-1}\bigr{)} a.s. and $U_{2}\,=\,\sup_{i\in\mathbb{N}}Z_{i}^{2}$ a.s. Hence

[TABLE]

So  \sum_{i=1}^{\infty}i^{-2}{\rm E}\bigl{(}(Z_{i}e_{i})^{2}\big{|}{\cal F}_{i-1}\bigr{)}\ <\infty\ \mbox{ a.s.} and the result follows from Theorem 2.18 of Hall and Heyde [8].

(c)  Fix any $\alpha>0$. By compactness of ${\cal X}\times\Theta$ and continuity of $h$ there exist a finite number $q\in\mathbb{N}$ and nonempty, pairwise disjoint, and measurable subsets $R_{1},\ldots,R_{q}$ of ${\cal X}$ such that $\bigcup_{j=1}^{q}R_{j}={\cal X}$ and  $|h(x,\theta)-h(z,\theta)|\leq\alpha$  for all $x,z\in R_{j}$ and all $\theta\in\Theta$, $1\leq j\leq q$. Choose any points $z_{j}^{(0)}\in R_{j}$, $1\leq j\leq q$, and denote $c_{j}(\theta)=h(z_{j}^{(0)},\theta)$, $1\leq j\leq q$, $\theta\in\Theta$. Then

[TABLE]

Introduce zero-one-valued random variables  Z_{i}^{(j)}:=\hskip 1.00006pt1\hskip-6.00006pt1\hskip 1.00006pt\bigl{(}X_{i}^{-1}(R_{j})\bigr{)}, $i\in\mathbb{N}$, $1\leq j\leq q$, i.e., $Z_{i}^{(j)}$ yields the value $1$ if the value of $X_{i}$ is in $R_{j}$, and otherwise $Z_{i}^{(j)}$ yields the value [math] . Abbreviate $W_{n}(\theta):=\sum_{i=1}^{n}h(X_{i},\theta)\,e_{i}$. Clearly,  $W_{n}(\theta)=\sum_{j=1}^{q}\sum_{i=1}^{n}h(X_{i},\theta)\,Z_{i}^{(j)}e_{i}$, and by (3.1) for all $i$, $j$, and $\theta$,

[TABLE]

where ${\rm sgn}(t):=1$ if $t\geq 0$ and ${\rm sgn}(t):=-1$ if $t<0$, for any real number $t$. Hence by summation in (3.2) over $i$ and $j$,

[TABLE]

Denote  $\overline{c}:=\sup_{(x,\theta)\in{\cal X}\times\Theta}|h(x,\theta)|$, which is finite and, clearly, $|c_{j}(\theta)|\leq\overline{c}$ for all $j=1,\ldots,q$ and all $\theta\in\Theta$. Hence

[TABLE]

and thus

[TABLE]

Applying parts (a) and (b) of the lemma,

[TABLE]

where  $U\,:=\,\limsup_{n\to\infty}\frac{1}{n}\sum_{i=1}^{n}|e_{i}|$, which is almost surely finite. Since $\alpha>0$ was arbitrary the result follows. $\Box$

3.2 Adaptive GLM and ML-estimators

Now we specialize to an ‘adaptive generalized linear model’ as follows. The parameter space $\Theta$ is a compact subset of $\mathbb{R}^{p}$ provided with the usual Euclidean metric, the mean response function $\mu$ is of the form

[TABLE]

where $f\,:\,{\cal X}\longrightarrow\mathbb{R}^{p}$ is a given continuous function whose range $f({\cal X})$ spans $\mathbb{R}^{p}$ and $G\,:\,I\longrightarrow\mathbb{R}$ is a given continuously differentiable function on an open interval $I\subseteq\mathbb{R}$ with \bigl{\{}f^{\sf\scriptsize T}(x)\,\theta\,:\,(x,\theta)\in{\cal X}\times\Theta\bigr{\}}\subseteq I and whose derivative $G^{\prime}$ is positive, $G^{\prime}(u)>0$ for all $u\in I$. The function $G$ is the inverse of the link function of the generalized linear model and $f^{\sf\scriptsize T}(x)\,\theta$, $(x,\theta)\in{\cal X}\times\Theta$, is the linear predictor. Note that an interval may be unbounded from below or from above or both, where in the latter case the interval is the whole real line. Assumption (A2) is strengthened by an assumption (A2’) below, stating that the conditional distribution of $Y_{i}$ given ${\cal F}_{i-1}$ belongs to a one-parameter exponential family of distributions $P_{\tau}$, $\tau\in J$, where $J\subseteq\mathbb{R}$ is an open interval. We employ the canonical (or ‘natural’) parametrization of the one-parameter exponential family where $\tau$ is its canonical parameter. So $P_{\tau}$, $\tau\in J$, are probability distributions on the Borel sigma-field of the real line with densities w.r.t. some Borel-measure $\nu$,

[TABLE]

where $K$ is a nonnegative measurable function on $\mathbb{R}$ and $b$ is a real-valued function on $J$, which is infinitely often differentiable, see e.g. Fahrmeir and Kaufmann [5], Section 2. In particular, the first and second derivatives of $b$ give the expectation and the variance of the distribution $P_{\tau}$, resp., $b^{\prime}(\tau)={\rm E}_{P_{\tau}}(Y)$ and $b^{\prime\prime}(\tau)={\rm Var}_{P_{\tau}}(Y)\,>0$  for all $\tau\in J$. So the derivative $b^{\prime}$ is a smooth and strictly increasing function and hence a bijection, $b^{\prime}\,:\,J\longrightarrow M$ where $M$ is the open interval of all expectations ${\rm E}_{P_{\tau}}(Y)$, $\tau\in J$. The inverse $(b^{\prime})^{-1}$ assigns to each expectation $m\in M$ the parameter value $\tau=(b^{\prime})^{-1}(m)\in J$ of the exponential family.

Now, we assume the following (A2’) which is stronger than assumption (A2) from Subsection 3.1. Recall that $\overline{\theta}$ denotes the fixed true parameter point.

• (A2’)

The values of the inverse link function $G$ are contained in $M$, i.e., $G(I)\subseteq M$.

For each $i\in\mathbb{N}$ the conditional distribution of $Y_{i}$ given ${\cal F}_{i-1}$ is equal to $P_{\overline{\tau}_{i}}$ where

\overline{\tau}_{i}=(b^{\prime})^{-1}\bigl{(}G(f^{\sf\scriptsize T}(X_{i})\,\overline{\theta})\bigr{)}.

For the notion of a conditional distribution of a real-valued random variable given a sub-sigma-field we refer to [3], p. 77, Definition 4.29. Note that $P_{\tau}$ has finite moments m_{k}(\tau)={\rm E}_{P_{\tau}}\bigl{(}Y^{k}\bigr{)} of any order $k=1,2,\ldots$, and $m_{k}(\tau)$ is a continuous function of $\tau\in J$. Assumption (A2’) together with (A1) imply the following. Firstly, {\rm E}\bigl{(}Y_{i}\big{|}\,{\cal F}_{i-1}\bigr{)}=m_{1}(\overline{\tau}_{i})=G\bigl{(}f^{\sf\scriptsize T}(X_{i})\,\overline{\theta}\bigr{)}. So, e_{i}=Y_{i}-G\bigl{(}f^{\sf\scriptsize T}(X_{i})\,\overline{\theta}\bigr{)}, $i\in\mathbb{N}$, satisfy  $Y_{i}=\mu(X_{i},\overline{\theta})+e_{i}$ and {\rm E}\bigl{(}e_{i}\big{|}\,{\cal F}_{i-1}\bigr{)}=0, with $\mu$ from (3.3). Secondly, {\rm E}\bigl{(}e_{i}^{2}\big{|}\,{\cal F}_{i-1}\bigr{)}=m_{2}(\overline{\tau}_{i})-\bigl{(}m_{1}(\overline{\tau}_{i})\bigr{)}^{2} for all $i\in\mathbb{N}$, and since the values of all $\overline{\tau}_{i}$ are contained in some compact subinterval of $J$ one has  {\rm E}\bigl{(}e_{i}^{2}\big{|}\,{\cal F}_{i-1}\bigr{)}\leq C_{2} a.s. for all $i\in\mathbb{N}$ for some real constant $C_{2}>0$. A similar conclusion holds for higher conditional moments of $e_{i}$, e.g. consider fourth moments:

[TABLE]

where $m_{0}(\overline{\tau}_{i})=1$ and G^{0}\bigl{(}f^{\sf\scriptsize T}(X_{i})\,\overline{\theta}\bigr{)}=1. The values of all the $X_{i}$, $i\in\mathbb{N}$, are in the compact experimental region ${\cal X}$, and hence all the random variables $\overline{\tau}_{i}$, $i\in\mathbb{N}$, have their values in some compact subinterval of $J$. It follows that

[TABLE]

for some real constant $C_{4}>0$. To summarize: assumption (A2’) together with (A1) imply (A2) and, moreover, (3.5). Obviously, this is due to the compactness of the experimental region ${\cal X}$ (and the continuity of $f$). Compactness of the parameter space $\Theta$, however, is not needed here since (A2’) as well as (A2) are local conditions at the true parameter point $\overline{\theta}$.

Fisher information matrices in a generalized linear model with univariate response whose observations follow a one-parameter exponential family were derived in Atkinson and Woods [2], formula (13.3) on p. 473, and also for the multivariate case in Fahrmeir and Kaufmann [5], p. 347. Accordingly, we employ the following assumption (A3’) on the family of functions $f_{\theta}$, $\theta\in\Theta$, defining the information matrices of designs via (1.1).

• (A3’)

f_{\theta}(x)\,=\,\ \varphi\bigl{(}f^{\sf\scriptsize T}(x)\,\theta\bigr{)}\,f(x)  for all $x\in{\cal X}$, $\theta\in\Theta$,

where  \varphi(u)\,=\,G^{\prime}(u)\Big{/}\sqrt{b^{\prime\prime}\Bigl{(}(b^{\prime})^{-1}\bigl{(}G(u)\bigr{)}\Bigr{)}},  $u\in I$,

and where $f$ is a given continuous $\mathbb{R}^{p}$-valued function on ${\cal X}$ whose range $f({\cal X})$ spans $\mathbb{R}^{p}$. In particular, by (A3’) the family $f_{\theta}$, $\theta\in\Theta$, satisfies condition (GLM) from Section 2.

In what follows we focus on the asymptotics of adaptive maximum likelihood (ML) estimators. Note, however, that the adaptive estimators $\widehat{\theta}_{n}$, $n\geq n_{\rm\scriptsize st}$, employed by the algorithm may or may not be given by the adaptive ML-estimators $\widehat{\theta}_{n}^{(\rm\scriptsize ML)}$, $n\geq n_{\rm\scriptsize st}$. The algorithm may employ any reasonable adaptive estimators $\widehat{\theta}_{n}$, $n\geq n_{\rm\scriptsize st}$, e.g., the adaptive maximum quasi-likelihood estimators studied by Chen, Hu, and Ying [4] in the case that the function $G$ is defined on the whole real line, $I=\mathbb{R}$. See also our remark below following Corollary 3.2. The main topics studied are strong consistency of the adaptive ML-estimators, i.e., almost-sure convergence to the true parameter point $\overline{\theta}$, and asymptotic normality. Strong consistency of the estimators $\widehat{\theta}_{n}$, $n\geq n_{\rm\scriptsize st}$, employed by the algorithm implies almost-sure asymptotic local D-optimality at $\overline{\theta}$ of the design sequence $\xi_{n}$ generated by the algorithm, which is an immediate consequence from Theorem 2.8. Note that the corollary does not need any of the assumptions (A1), (A2), (A2’), or (A3’).

Corollary 3.2

If $\widehat{\theta}_{n}\,\stackrel{{\scriptstyle\rm\scriptsize a.s.}}{{\longrightarrow}}\,\overline{\theta}$ then for any sequence $\widehat{\theta}_{n}^{\prime}$ of estimators such that $\widehat{\theta}_{n}^{\prime}\,\stackrel{{\scriptstyle\rm\scriptsize a.s.}}{{\longrightarrow}}\,\overline{\theta}$ one has  $M(\xi_{n},\widehat{\theta}_{n}^{\prime})\,\stackrel{{\scriptstyle\rm\scriptsize a.s.}}{{\longrightarrow}}\,M(\xi^{*}_{\overline{\theta}},\overline{\theta})$, where $\xi^{*}_{\overline{\theta}}$ is a locally D-optimal design at $\overline{\theta}$.

Remark.  Under assumptions (A1), (A2’), and (A3’), in the case $I=\mathbb{R}$ the adaptive maximum quasi-likelihood estimators studied by Chen, Hu, and Ying [4] turn out to be strongly consistent. In fact, by Corollary 2.7, part (ii), and by (3.5) one easily verifies the assumptions of Theorem 2 in [4] for the adaptive design sequence generated by the algorithm, irrespective of the employed sequence of adaptive estimators $\widehat{\theta}_{n}$, $n\geq n_{\rm\scriptsize st}$ in the algorithm. Our next result establishes strong consistency of the adaptive maximum likelihood estimators, again irrespective of the employed sequence of estimators $\widehat{\theta}_{n}$, $n\geq n_{\rm\scriptsize st}$ in the algorithm. $\Box$

Assuming (A1) and (A2’), an adaptive ML-estimator $\widehat{\theta}_{n}^{(\rm\scriptsize ML)}=\widehat{\theta}_{n}^{(\rm\scriptsize ML)}(X_{1},Y_{1},\ldots,X_{n},Y_{n})$, for $n\geq n_{\rm\scriptsize st}$, is a maximizer of the log-likelihood

[TABLE]

Note that with probability equal to one, $K(Y_{i})>0$ for all $i\in\mathbb{N}$. Thus positivity of $K(Y_{i})$, $i\in\mathbb{N}$, is assumed for the log-likelihood (3.6). Note also that for the canonical link one gets $b^{\prime}=G$ and hence $\tau_{i}(\theta)=f^{\sf\scriptsize T}(X_{i})\,\theta$. The following result gives the strong consistency of the adaptive ML-estimators.

Theorem 3.3

Under assumptions (A1), (A2’), and (A3’), one has  $\widehat{\theta}_{n}^{(\rm\scriptsize ML)}\,\stackrel{{\scriptstyle\rm\scriptsize a.s.}}{{\longrightarrow}}\,\overline{\theta}$.

Proof.  For all $\theta\in\Theta$ one gets from (3.6) and (3.7), observing that $\tau_{i}(\overline{\theta})=\overline{\tau}_{i}$ and Y_{i}=G\bigl{(}f^{\sf\scriptsize T}(X_{i})\,\overline{\theta}\bigr{)}+e_{i},

[TABLE]

For each $i\in\mathbb{N}$, by second order Taylor expansion of $b(\tau)$ at $\overline{\tau}_{i}$,

[TABLE]

with some $\widetilde{\tau}_{i}(\theta)$ from the interval whose end points are given by $\overline{\tau}_{i}$ and $\tau_{i}(\theta)$. Since  $b^{\prime}(\overline{\tau}_{i})=G(f^{\sf\scriptsize T}(X_{i})\,\overline{\theta})$, (3.8) rewrites as

[TABLE]

By compactness of ${\cal X}$ and $\Theta$ and continuity of $f$ there is a compact subinterval $[c_{1},c_{2}]\subseteq I$ such that $c_{1}\leq f^{\sf\scriptsize T}(x)\,\theta\leq c_{2}$ for all $x\in{\cal X}$, $\theta\in\Theta$. Since $G$ and $(b^{\prime})^{-1}$ are increasing functions,  d_{1}\leq(b^{\prime})^{-1}\bigl{(}G(f^{\sf\scriptsize T}(x)\,\theta)\bigr{)}\leq d_{2}  for all $x\in{\cal X}$, $\theta\in\Theta$, where d_{j}=(b^{\prime})^{-1}\bigl{(}G(c_{j})\bigr{)}, $j=1,2$, and $[d_{1}\,,\,d_{2}]\subseteq J$. In particular, $d_{1}\leq\tau_{i}(\theta)\leq d_{2}$ for all $i$ and $\theta$. By continuity and positivity of $b^{\prime\prime}$ the minimum $\beta_{0}=\min_{d_{1}\leq\tau\leq d_{2}}b^{\prime\prime}(\tau)$ exists and $\beta_{0}>0$. Hence, in (3.9), b^{\prime\prime}\bigl{(}\widetilde{\tau}_{i}(\theta)\bigr{)}\geq\beta_{0} for all $i$ and $\theta$. Since the composition (b^{\prime})^{-1}\bigl{(}G(u)\bigr{)}, $u\in I$, is a continuously differentiable function with positive derivative H(u)\,=\,\frac{\rm d}{{\rm d}u}\bigl{[}(b^{\prime})^{-1}\bigl{(}G(u)\bigr{)}\bigr{]}, $u\in I$, it follows that  $\beta_{1}\,=\,\min_{c_{1}\leq u\leq c_{2}}H(u)$  exists and $\beta_{1}>0$. By the mean value theorem

[TABLE]

From (3.7) and (3.9) it follows that

[TABLE]

As in Wu [15], Lemma 1, strong consistency of $\widehat{\theta}_{n}^{\rm\scriptsize(ML)}$ will follow if we prove that for every $\delta>0$ such that the parameter subset C(\overline{\theta},\delta)=\bigl{\{}\theta\in\Theta\,:\,\|\theta-\overline{\theta}\|\geq\delta\bigr{\}} is nonempty, one has

[TABLE]

In fact, the $\liminf$ turns out to be equal to infinity almost surely, since we show that

[TABLE]

By (3.10) and the trivial inequality $a\geq-|a|$ for all real $a$,

[TABLE]

Since  $\overline{\tau}_{i}-\tau_{i}(\theta)=h(X_{i},\theta)$ for all $i\in\mathbb{N}$ and $\theta\in\Theta$, with h(x,\theta)=(b^{\prime})^{-1}\bigl{(}f^{\sf\scriptsize T}(x)\,\overline{\theta}\bigr{)}-(b^{\prime})^{-1}\bigl{(}f^{\sf\scriptsize T}(x)\,\theta\bigr{)}, $(x,\theta)\in{\cal X}\times\Theta$, it follows by Lemma 3.1, part (c), that

[TABLE]

It remains to show that the $\liminf$ of the second term on the r.h.s. of (3.12) is positive almost surely. Consider an arbitrary path of the adaptive process and, in particular, a path $x_{i}$, $i\in\mathbb{N}$, of the sequence of random variables $X_{i}$, $i\in\mathbb{N}$. With the generated design sequence $\xi_{n}$, $n\geq n_{\rm\scriptsize st}$, we can write, for all $n\geq n_{\rm\scriptsize st}$,

[TABLE]

For any $\theta\in\Theta$, $\theta\not=\overline{\theta}$, denote $c_{\theta}=(\theta-\overline{\theta})/\|\theta-\overline{\theta}\|$ and V_{p-1,\theta}=\bigl{\{}a\in\mathbb{R}^{p}\,:\,c^{\sf\scriptsize T}_{\theta}a=0\bigr{\}}. By Theorem 2.6 there exist $n_{0}\geq n_{\rm\scriptsize st}$, $\varepsilon>0$, and $\alpha\in(\,0\,,\,1\,)$ such that

[TABLE]

Using (A3’) and $f^{\sf\scriptsize T}(x)\,\theta\in[\,c_{1}\,,\,c_{2}\,]\subseteq I$ for all $(x,\theta)\in{\cal X}\times\Theta$, let

[TABLE]

hence $0<\varphi_{\rm\scriptsize min}\leq\varphi_{\rm\scriptsize max}<\infty$. Define  $\varepsilon^{\prime}:=\varepsilon\,\varphi_{\rm\scriptsize min}/\varphi_{\rm\scriptsize max}$. As in the proof of Lemma 2.5, part (i), one gets

[TABLE]

Note that  f^{-1}\bigl{(}\overline{V}_{p-1,\theta}(\varepsilon^{\prime})\bigr{)}\,=\,\bigl{\{}x\in{\cal X}\,:\,|c^{\sf\scriptsize T}_{\theta}f(x)|\leq\varepsilon^{\prime}\bigr{\}}.  Taking the complementary sets and observing that $|c^{\sf\scriptsize T}_{\theta}f(x)|>\varepsilon^{\prime}$ is equivalent to \big{|}f^{\sf\scriptsize T}(x)\,(\theta-\overline{\theta})\big{|}>\varepsilon^{\prime}\|\theta-\overline{\theta}\|, which in the case $\theta\in C(\overline{\theta},\delta)$ implies \big{|}f^{\sf\scriptsize T}(x)\,(\theta-\overline{\theta})\big{|}>\varepsilon^{\prime}\delta, we have

[TABLE]

Hence

[TABLE]

Together with (3.14), (3.13), and (3.12) the proof of (3.11) is complete and $\widehat{\theta}_{n}^{(\rm\scriptsize ML)}\,\stackrel{{\scriptstyle\rm\scriptsize a.s.}}{{\longrightarrow}}\,\overline{\theta}$ follows. $\Box$

The next result shows the asymptotic normality of adaptive ML-estimators if the true parameter point $\overline{\theta}$ is an interior point of the parameter space $\Theta$, i.e., there exists a $\rho>0$ such that

[TABLE]

The $p$-dimensional normal distribution with expectation [math] and (positive definite) covariance matrix $C$ is denoted by ${\rm N}(0,C)$. For $C=I_{p}$, the $p\times p$ identity matrix, ${\rm N}(0,I_{p})$ is the $p$-dimensional standard normal distribution. For a sequence $W_{n}$ of $\mathbb{R}^{p}$-valued random variables, convergence in distribution of $W_{n}$ (as $n\to\infty$) to a $p$-dimensional normal distribution ${\rm N}(0,C)$ is abbreviated by $W_{n}\,\stackrel{{\scriptstyle\rm\scriptsize d}}{{\longrightarrow}}\,{\rm N}(0,C)$. In the next theorem, the assumption of strong consistency of the adaptive estimators $\widehat{\theta}_{n}$ employed by the algorithm is met, by Theorem 3.3, if $\widehat{\theta}_{n}=\widehat{\theta}_{n}^{\rm\scriptsize(ML)}$, $n\geq n_{\rm\scriptsize st}$.

Theorem 3.4

Assume (A1), (A2’), and (A3’). Assume further that $G$ is twice continuously differentiable, $\overline{\theta}$ is an interior point of $\Theta$, and $\widehat{\theta}_{n}\,\stackrel{{\scriptstyle\rm\scriptsize a.s.}}{{\longrightarrow}}\,\overline{\theta}$. Then:

[TABLE]

Also, denoting by M_{*}=M\bigl{(}\xi^{*}_{\overline{\theta}},\overline{\theta}) the information matrix of a locally D-optimal design at $\overline{\theta}$, one has

[TABLE]

Proof.  Choose a positive $\rho_{1}<\rho$ where $\rho>0$ is according to (3.15). Then the compact ball \overline{B}(\overline{\theta},\rho_{1})=\bigl{\{}a\in\mathbb{R}^{p}\,:\,\|a-\overline{\theta}\|\leq\rho_{1}\bigr{\}} is contained in the interior of $\Theta$. By Theorem 3.3 $\widehat{\theta}_{n}^{\rm\scriptsize(ML)}\in\overline{B}(\overline{\theta},\rho_{1})\ \mbox{ a.s.}$  if $n$ is large enough, i.e., if $n$ is greater than or equal to the value of some random variable $N$ whose values are in $\mathbb{N}$. Let $n\geq n_{\rm\scriptsize st}$ be given and denote by $\{N\leq n\}$ the event (subset of $\Omega$) that the random variable $N$ yields a value less than or equal to $n$. Consider the log-likelihood function $L_{n}(\theta)$ from (3.6), (3.7) and its gradient $S_{n}(\theta)=\nabla L_{n}(\theta)$ w.r.t. $\theta$, which is often called the score function, for $\theta\in\overline{B}(\overline{\theta},\rho_{1})$. One obtains

[TABLE]

Abbreviate

[TABLE]

Since S_{n}\bigl{(}\widehat{\theta}_{n}^{\rm\scriptsize(ML)}\bigr{)}=0  a.s. on $\{N\leq n\}$  we get from (3.16),

[TABLE]

The function $H$ from (3.17) is continuously differentiable. Denote its derivative by $H^{\prime}$. The gradient (w.r.t. $\theta$) of $R_{i}(\theta)$ is given by

[TABLE]

By the mean value theorem, for each $i=1,\ldots,n$ there is some $\widetilde{\theta}_{i,n}$ on the line segment joining $\overline{\theta}$ and $\widehat{\theta}_{n}^{\rm\scriptsize(ML)}$ such that

[TABLE]

Together with (3.18) and (3.19), and observing that  $G^{\prime}\,H\,=\,\varphi^{2}$, we get

[TABLE]

Since M(\xi_{n},\theta)=\frac{1}{n}\sum_{i=1}^{n}\varphi^{2}\bigl{(}f^{\sf\scriptsize T}(X_{i})\,\theta\bigr{)}\,f(X_{i})\,f^{\sf\scriptsize T}(X_{i}) we can write

[TABLE]

So, by (3.20) after some slight manipulations,

[TABLE]

Next we show that

[TABLE]

Regarding the Cramér-Wold device let any $v\in\mathbb{R}^{p}$ with $\|v\|=1$ be given. Using (3.16) for $\theta=\overline{\theta}$ and inserting Y_{i}=G\bigl{(}f^{\sf\scriptsize T}(X_{i})\,\overline{\theta}\bigr{)}\,+e_{i} we can write

[TABLE]

Clearly, for each $i\in\mathbb{N}$ the random variable Z_{i}:=H\bigl{(}f^{\sf\scriptsize T}(X_{i})\,\overline{\theta}\bigr{)}\,v^{\sf\scriptsize T}M_{*}^{-1/2}f(X_{i}) is ${\cal F}_{i-1}$-measurable and $|Z_{i}|\leq c$ for all $i\in\mathbb{N}$ for some finite constant $\widetilde{c}$. Since $\widetilde{e}_{i}=e_{i}Z_{i}$, $i\in\mathbb{N}$, one easily verifies that the sequence of partial sums $\sum_{i=1}\widetilde{e}_{i}$, $n\in\mathbb{N}$, is a martingale w.r.t. ${\cal F}_{n}$, $n\in\mathbb{N}$. By Corollary 3.1 (p. 58) in [8] the convergence  $\frac{1}{\sqrt{n}}\,\sum_{i=1}^{n}\widetilde{e}_{i}\,\stackrel{{\scriptstyle\rm\scriptsize d}}{{\longrightarrow}}\,{\rm N}(0,1)$  holds if the following two conditions (A) and (B) are satisfied.

• (A)

\displaystyle\frac{1}{n}\sum_{i=1}^{n}{\rm E}\bigl{(}\widetilde{e}_{i}^{2}\big{|}\,{\cal F}_{i-1}\bigr{)}\,\,\stackrel{{\scriptstyle\rm\scriptsize a.s.}}{{\longrightarrow}}\,1.

• (B)

\displaystyle\frac{1}{n}\sum_{i=1}^{n}{\rm E}\Bigl{(}\widetilde{e}_{i}^{2}\hskip 1.00006pt1\hskip-6.00006pt1\hskip 1.00006pt\bigl{(}|\widetilde{e}_{i}|>\sqrt{n}\,\varepsilon\bigr{)}\big{|}\,{\cal F}_{i-1}\Bigr{)}\,\,\stackrel{{\scriptstyle\rm\scriptsize a.s.}}{{\longrightarrow}}\,0  for all $\varepsilon>0$.

Ad (A).  {\rm E}\bigl{(}\widetilde{e}_{i}^{2}\big{|}\,{\cal F}_{i-1}\bigr{)}={\rm E}\bigl{(}e_{i}^{2}\big{|}\,{\cal F}_{i-1}\bigr{)}\,Z_{i}^{2} and Z_{i}^{2}\,=\,H^{2}\bigl{(}f^{\sf\scriptsize T}(X_{i})\,\overline{\theta}\bigr{)}\,v^{\sf\scriptsize T}M_{*}^{-1/2}\,f(X_{i})\,f^{\sf\scriptsize T}(X_{i})\,M_{*}^{-1/2}\,v. By assumption (A2’), {\rm E}\bigl{(}e_{i}^{2}\big{|}\,{\cal F}_{i-1}\bigr{)}=b^{\prime\prime}\Bigl{(}(b^{\prime})^{-1}\bigl{(}G(f^{\sf\scriptsize T}(X_{i})\,\overline{\theta})\bigr{)}\Bigr{)}. Together with (3.17) this yields  {\rm E}\bigl{(}\widetilde{e}_{i}^{2}\big{|}\,{\cal F}_{i-1}\bigr{)}=\varphi^{2}\bigl{(}f^{\sf\scriptsize T}(X_{i})\,\overline{\theta}\bigr{)}\,v^{\sf\scriptsize T}M_{*}^{-1/2}\,f(X_{i})\,f^{\sf\scriptsize T}(X_{i})\,M_{*}^{-1/2}v, and hence

[TABLE]

where the convergence follows from Corollary 3.2.

Ad (B). Using the trivial inequality  \widetilde{e}_{i}^{2}\,\hskip 1.00006pt1\hskip-6.00006pt1\hskip 1.00006pt\bigl{(}|\widetilde{e}_{i}|>\sqrt{n}\,\varepsilon\bigr{)}\,\leq\,\frac{1}{\varepsilon^{2}n}\,\widetilde{e}_{i}^{4}  we obtain

[TABLE]

where we have used (3.5).

So by the Cramér-Wold device (3.26) follows and hence, by (3.25),

[TABLE]

Next we show that

[TABLE]

By (3.24),

[TABLE]

Note that  \big{\|}f(X_{i})\,f^{\sf\scriptsize T}(X_{i})\big{\|}=\big{\|}f(X_{i})\big{\|}^{2}\leq\gamma_{0}^{2} for all $i\in\mathbb{N}$, where $\gamma_{0}=\sup_{x\in{\cal X}}\|f(x)\|<\infty$. By $f^{\sf\scriptsize T}(X_{i})\,\theta\in[\,c_{1}\,,\,c_{2}\,]\subseteq I$ for all $i\in\mathbb{N}$ and $\theta\in\overline{B}(\overline{\theta},\rho_{1})$ for some compact interval $[\,c_{1}\,,\,c_{2}\,]$, by the uniform continuity of the function $\varphi^{2}$ on the compact interval, and by

[TABLE]

it follows that

[TABLE]

From this the first convergence statement in (3.28) follows. To prove the second convergence statement in (3.28) we write Y_{i}-G\bigl{(}f^{\sf\scriptsize T}(X_{i})\,\widetilde{\theta}_{i,n}\bigr{)}=G\bigl{(}f^{\sf\scriptsize T}(X_{i})\,\overline{\theta}\bigr{)}-G\bigl{(}f^{\sf\scriptsize T}(X_{i})\,\widetilde{\theta}_{i,n}\bigr{)}+e_{i}, and together with the definition of $B_{n}$ in (3.22),

[TABLE]

One concludes  $\frac{1}{n}B_{n}^{(1)}\,\stackrel{{\scriptstyle\rm\scriptsize a.s.}}{{\longrightarrow}}\,0$  by similar arguments as used above when showing $\frac{1}{n}D_{n}\,\stackrel{{\scriptstyle\rm\scriptsize a.s.}}{{\longrightarrow}}\,0$ where, in particular, uniform continuity of $G$ on the compact interval $[\,c_{1}\,,\,c_{2}\,]\subseteq I$ and boundedness of $H^{\prime}$ on that interval are utilized. Consider the sequence of matrices $\frac{1}{n}B_{n}^{(2)}$ entrywise. The $(k,\ell)$-th entry (where $1\leq k,\ell\leq p$) of $\frac{1}{n}B_{n}^{(2)}$ has the form  $\frac{1}{n}\sum_{i=1}^{n}Z_{i}\,e_{i}$ where  Z_{i}=H^{\prime}\bigl{(}f^{\sf\scriptsize T}(X_{i})\,\overline{\theta}\bigr{)}\,f_{k}(X_{i})\,f_{\ell}(X_{i}). Note that $|Z_{i}|\leq c$ for all $i\in\mathbb{N}$ for some real constant $c>0$. From Lemma 3.1, part (b), it follows that  $\frac{1}{n}\sum_{i=1}^{n}Z_{i}\,e_{i}\,\,\stackrel{{\scriptstyle\rm\scriptsize a.s.}}{{\longrightarrow}}\,0$. Hence  $\frac{1}{n}B_{n}^{(2)}\,\stackrel{{\scriptstyle\rm\scriptsize a.s.}}{{\longrightarrow}}\,0$. For $\frac{1}{n}B_{n}^{(3)}$ we consider again the $(k,\ell)$-th entry for any given $1\leq k,\ell\leq p$, and we obtain

[TABLE]

The uniform continuity of $H^{\prime}$ on $[\,c_{1}\,,\,c_{2}\,]$ yields  \max_{1\leq i\leq n}\big{|}H^{\prime}\bigl{(}f^{\sf\scriptsize T}(X_{i})\,\widetilde{\theta}_{i,n}\bigr{)}-H^{\prime}\bigl{(}f^{\sf\scriptsize T}(X_{i})\,\overline{\theta}\bigr{)}\big{|}\,\stackrel{{\scriptstyle\rm\scriptsize a.s.}}{{\longrightarrow}}\,0. Together with Lemma 3.1, part (a), it follows that the $(k,\ell)$-th entry of $\frac{1}{n}B_{n}^{(3)}$ converges to zero almost surely and hence $\frac{1}{n}B_{n}^{(3)}\,\stackrel{{\scriptstyle\rm\scriptsize a.s.}}{{\longrightarrow}}\,0$. So we have proved (3.28). By Corollary 3.2, $M(\xi_{n},\overline{\theta})\,\stackrel{{\scriptstyle\rm\scriptsize a.s.}}{{\longrightarrow}}\,M_{*}$ and hence

[TABLE]

Together with (3.27), observing \lim_{n\to\infty}\mathbb{P}_{\overline{\theta}}\bigl{(}\{N\leq n\}\bigr{)}=1 and using standard properties of convergence in distribution, it follows that for any sequence of $p\times p$ random matrices $Q_{n}$ such that $Q_{n}\,\stackrel{{\scriptstyle\rm\scriptsize a.s.}}{{\longrightarrow}}\,M_{*}^{1/2}$ one has  Q_{n}\bigl{[}\sqrt{n}\bigl{(}\widehat{\theta}_{n}^{\rm\scriptsize(ML)}-\overline{\theta}\bigr{)}\bigr{]}\,\stackrel{{\scriptstyle\rm\scriptsize d}}{{\longrightarrow}}\,{\rm N}(0,I_{p}). In particular, the convergence holds for the sequence $Q_{n}=M^{1/2}(\xi_{n},\widehat{\theta}_{n}^{\rm\scriptsize(ML)})$ and the constant sequence $Q_{n}=M_{*}^{1/2}$. Hence the result follows. $\Box$

Appendix A Appendix: known auxiliary results

Four well-known results on nonnegative definite matrices are stated below, which have been used throughout the proofs. If $A$ is a (real) $p\times q$ matrix then the range of $A$ is given by ${\rm range}(A)=\{Az\,:\,z\in\mathbb{R}^{q}\}$. A generalized inverse of a $p\times q$ matrix $A$ is denoted by $A^{-}$ which is by definition a $q\times p$ matrix satisfying $AA^{-}A=A$. As it is easily seen, if $A$ is symmetric $p\times p$ and $b\in{\rm range}(A)$ then the value $b^{\sf\scriptsize T}A^{-}b$ is the same for all generalized inverses of $A$.

• (M1)

If $A$ and $B$ are nonnegative definite $p\times p$ matrices and $A\leq B$, then  ${\rm range}(A)\subseteq{\rm range}(B)$  and  $z^{\sf\scriptsize T}A^{-}z\,\geq z^{\sf\scriptsize T}B^{-}z$  for all $z\in{\rm range}(A)$. See Stepniak, Wang and Wu [13], Lemma 2.

• (M2)

If $a_{1},\ldots,a_{r}\in\mathbb{R}^{p}$ and $\lambda_{1},\ldots,\lambda_{r}\in(\,0\,,\,\infty)$ such that $\sum_{j=1}^{r}\lambda_{j}=1$, then

[TABLE]

The first inequality is a special case of Theorem 4.2 in Gaffke and Krafft [7]; the second inequality follows from $a^{\sf\scriptsize T}(aa^{\sf\scriptsize T})^{-}a=1$ if $a\not=0$, and $a^{\sf\scriptsize T}(aa^{\sf\scriptsize T})^{-}a=0$ if $a=0$, for any $a\in\mathbb{R}^{p}$.

• (M3)

If $M$ is a positive definite $p\times p$ matrix and $v\in\mathbb{R}^{p}$, $v\not=0$, then

[TABLE]

and the maximum on the right hand side is attained for  $b_{0}\,=\,M^{-1}v/(v^{\sf\scriptsize T}M^{-1}v)$. Actually, the inequality is a special case of a more general matrix inequality, see Pukelsheim [12], Section 1.21.

• (M4)

Let $A$ be a $p\times q$ matrix with columns $a_{1},\ldots,a_{q}\in\mathbb{R}^{p}$. Then

[TABLE]

and where ${\rm dist}^{2}(a,V)=\inf_{v\in V}\|a-v\|^{2}$ denotes the squared Euclidean distance of a vector $a\in\mathbb{R}^{p}$ and a linear subspace $V$ of $\mathbb{R}^{p}$. In fact, the formula trivially holds if the vectors $a_{1},\ldots,a_{q}$ are linearly dependent, in which case both sides of the formula are equal to zero. Also, the case $q=1$ is trivial. Let $q\geq 2$ and let $a_{1},\ldots,a_{q}$ be linearly independent. Consider the $p\times(q-1)$ matrix $B$ having columns $a_{1},\ldots,a_{q-1}$. Then, the matrix $A^{\sf\scriptsize T}A$ can be written in partitioned form as

[TABLE]

So, by a well-known formula for the determinant of a partitioned positive definite matrix,

[TABLE]

The second factor on the r.h.s. of the latter equation is equal to  ${\rm dist}^{2}(a_{q},V_{q-1})$. We have thus obtained that  \det\bigl{(}A^{\sf\scriptsize T}A\bigr{)}=\det\bigl{(}B^{\sf\scriptsize T}B)\cdot{\rm dist}^{2}(a_{q},V_{q-1}). Now the asserted formula follows by induction on $q$.