Estimable group effects for strongly correlated variables in linear models

TL;DR

This paper identifies linear combinations of strongly correlated variables in linear models that can be accurately estimated, revealing that the variability weighted average is uniquely estimable and more precise with stronger correlation.

Contribution

It introduces a method to find estimable linear combinations of parameters for strongly correlated predictors in linear models, highlighting the special role of the variability weighted average.

Findings

The variability weighted average is more accurately estimated with stronger correlation.

Such linear combinations can be computed easily in all linear models.

The approach has applications in inference and estimation for correlated variables.

Abstract

It is well known that parameters for strongly correlated predictor variables in a linear model cannot be accurately estimated. We look for linear combinations of these parameters that can be. Under a uniform model, we find such linear combinations in a neighborhood of a simple variability weighted average of these parameters. Surprisingly, this variability weighted average is more accurately estimated when the variables are more strongly correlated, and it is the only linear combination with this property. It can be easily computed for strongly correlated predictor variables in all linear models and has applications in inference and estimation concerning parameters of such variables.

Equations

$$\mathbf{y}=\mathbf{X}\mathbf{\beta }+\mathbf{\epsilon },$$

$$\mathbf{c}={\alpha }_1{\mathbf{v}}_1+{\alpha }_2{\mathbf{v}}_2+\cdots +{\alpha }_q{\mathbf{v}}_q.$$

$$var({\mathbf{c}}^T\hat{\mathbf{\beta }})/{\sigma }^2=\frac{{\alpha }_1^2}{{\lambda }_1}+\frac{{\alpha }_2^2}{{\lambda }_2}+\cdots +\frac{{\alpha }_q^2}{{\lambda }_q}.$$

$${\Xi }^{\prime }=\{\xi ({\mathbf{w}}^{\prime })|\xi ({\mathbf{w}}^{\prime })=w_1^{\prime }{\beta }_1+w_2^{\prime }{\beta }_2+\cdots +w_p^{\prime }{\beta }_p\},$$

$$\hat{\xi }({\mathbf{w}}^{\prime })\approx w_1^{\prime }{\beta }_1+w_2^{\prime }{\beta }_2+\cdots +w_p^{\prime }{\beta }_p.$$

$$\mathbf{y}=\mathbf{X}\mathbf{\beta }+\mathbf{\epsilon }$$

$$\mathbf{X}^{T}\mathbf{X}=\left(\begin{array}[]{ccccc}1&r_{12}&r_{13}&\dots&r_{1p}\\ r_{21}&1&r_{23}&\dots&r_{2p}\\ r_{31}&r_{32}&1&\dots&r_{3p}\\ \cdot&\cdot&\cdot&\dots&\cdot\\ \cdot&\cdot&\cdot&\dots&\cdot\\ r_{p1}&r_{p2}&r_{p3}&\dots&1\\ \end{array}\right)_{p\times p},$$

$$\hat{\mathbf{\beta }}=({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^T\mathbf{y},$$

$${\mathbf{x}}_i$$

$$|corr(\mathbf{x}_{i},\mathbf{x}_{1})|>\frac{\sqrt{2}}{2},\hskip 21.68121pt\mbox{for $i=2,3,\dots,p$}.$$

$${\mathbf{x}}_1,\text{sgn}(r_{12}){\mathbf{x}}_2,\dots ,\text{sgn}(r_{1p}){\mathbf{x}}_p.$$

$$\Xi =\{\xi (\mathbf{w})|\xi (\mathbf{w})=w_1{\beta }_1+w_2{\beta }_2+\cdots +w_p{\beta }_p\},$$

$$\hat{\xi }(\mathbf{w})={\mathbf{w}}^T\hat{\mathbf{\beta }}=w_1{\hat{\beta }}_1+w_2{\hat{\beta }}_2+\cdots +w_p{\hat{\beta }}_p,$$

$${\tau }_a=\xi ({\mathbf{w}}_0)={\mathbf{w}}_0^T\mathbf{\beta }=\frac{1}{p}\sum _{i=1}^p{\beta }_i.$$

$${\beta }_j=\xi ({\mathbf{w}}_j)$$

$$var(\hat{\xi }(\mathbf{w}))=\frac{[1+(p-2)r]{\sum }_{i=1}^p{w}_i^2-2r{\sum }_{i<j}{w}_i{w}_j}{1+(p-2)r-(p-1)r^2}.$$

$$var(\widehat{{\beta }_j},r)=var(\hat{\xi }({\mathbf{w}}_j),r)=\underset{\mathbf{w}}{\max }\{var(\hat{\xi }(\mathbf{w}),r)\}.$$

$$\underset{r\to 1}{\lim }var({\hat{\beta }}_j,r)=+\infty .$$

$$var({\hat{\tau }}_a,r)=var(\hat{\xi }({\mathbf{w}}_0),r)=\underset{\mathbf{w}}{\min }\{var(\hat{\xi }(\mathbf{w}),r)\}.$$

$$\underset{r\to 1}{\lim }var({\hat{\tau }}_a,r)=\frac{1}{p^2}.$$

$$\sum _{i=1}^p{w}_i^2=\sum _{i=1}^p(w_i^{\prime }{)}^2\text{mbox}but\sum _{i<j}^p{w}_i{w}_j\ge \sum _{i<j}^p{w}_i^{\prime }{w}_j^{\prime },$$

$$var(\hat{\xi }(\mathbf{w}))\le var(\hat{\xi }({\mathbf{w}}^{\prime })).$$

$$\mathbf{y}={\beta }_0{1}_n+\mathbf{X}\mathbf{\beta }+\mathbf{\epsilon },$$

$$E(\mathbf{\epsilon })=0\text{mbox}andvar(\mathbf{\epsilon })={\sigma }^2\mathbf{I}.$$

$$s_j={\left[\sum _{i=1}^n(x_{ij}-{\bar{x}}_j{)}^2\right]}^{1/2},$$

$$w_j^{\ast }=\frac{s_j}{{\sum }_{i=1}^p{s}_i}.$$

$${\tau }_w={\mathbf{w}}_w^T\mathbf{\beta }=\sum _{i=1}^p{w}_i^{\ast }{\beta }_i.$$

$${\hat{\tau }}_w={\mathbf{w}}_w^T\hat{\mathbf{\beta }}=\sum _{i=1}^p{w}_i^{\ast }{\hat{\beta }}_i.$$

$$\underset{r\to 1}{\lim }var({\hat{\tau }}_w,r)=\frac{{\sigma }^2}{({\sum }_{i=1}^p{s}_i{)}^2}.$$

$${\mathbf{x}}_1={\mathbf{z}}_1,{\mathbf{x}}_2=w_1{\mathbf{z}}_1+(1-w_1){\mathbf{z}}_2;$$

Full text

Estimable group effects for strongly correlated variables in linear models

MIN TSAO

Department of Mathematics & Statistics, University of Victoria

Abstract: When a linear model has a multicollinearity problem caused by a group of strongly correlated predictor variables, it is well known that parameters for variables in such a group cannot be accurately estimated. We study linear combinations of these parameters to look for ones that can be accurately estimated. Under a uniform model for which the level of multicollinearity can be quantified, we find such linear combinations in a neighbourhood of a simple variability weighted average of these parameters. This variability weighted average actually benefits from multicollinearity in that the variance of its minimum-variance unbiased linear estimator is a monotone decreasing function of the level of multicollinearity, so it can be more accurately estimated at higher levels of multicollinearity. It is the only linear combination with this property. It can be substantially more accurately estimated than the average of parameters of uncorrelated variables. We also demonstrate that it can be accurately estimated outside the uniform model. Accurately estimated linear combinations have applications in inference and estimation for parameters of strongly correlated variables. As an alternative to the Ridge regression, we propose a method which uses the variability weighted average to estimate such parameters while retaining the least-squares estimates for parameters of other variables in the model.

Key words: constrained local regression, estimable group effects, linear models, multicollinearity, ridge regression, strongly correlated predictor variables.

1 Introduction

Consider a linear model

[TABLE]

where $\mathbf{y}$ is an $n\times 1$ vector of observations, $\mathbf{X}=[\mathbf{1},\mathbf{x}_{1},\dots,\mathbf{x}_{q-1}]$ a known $n\times q$ design matrix, $\bm{\beta}=(\beta_{0},\beta_{1},\dots,\beta_{q-1})^{T}$ an unknown $q\times 1$ vector of regression parameters, and $\bm{\varepsilon}$ an $n\times 1$ vector of random errors with mean zero and variance matrix $\sigma^{2}\mathbf{I}$.

Suppose the first $p$ predictor variables $\mathbf{x}_{1},\mathbf{x}_{2},\dots,\mathbf{x}_{p}$ are strongly correlated ($2\leq p<q-1$). Then, this group of $p$ variables generates a multicollinearity problem for model (1). This type of multicollinearity problem arises often from data sets in observational studies where strongly correlated variables are not uncommon. It manifests numerically through unusually large variances of least-squares estimators for parameters of the $p$ variables, sometimes accompanied by estimated parameters with unusually large absolute values. Reasons behind these numerical observations are well documented. They can be found in many books on linear models, e.g., Draper and Smith (1998) and Belsley, Kuh and Welsch (2004). In particular, the latter book gives a comprehensive coverage on the detection of and remedies for the multicollinearity problem.

Although parameters of the strongly correlated variables $\beta_{1}$, $\beta_{2}$, $\dots,$ $\beta_{p}$ cannot be accurately estimated, other parameters in the model and some linear combinations of all $q$ parameters can still be. For convenience, we refer to a linear combination of $\beta_{0},\beta_{1},\dots,\beta_{q-1}$ as an effect of the underlying variables, and an effect is said to be estimable if it has an unbiased linear estimator whose variance is smaller than or comparable to the error variance $\sigma^{2}$, similar in magnitude to variances of least-squares estimators for parameters not affected by the multicollinearity problem. Silvey (1969) gave the following result describing those effects that may be and those that may not be accurately estimated. For simplicity, suppose $\mathbf{X}^{T}\mathbf{X}$ is nonsingular so that the least-squares estimator $\hat{\bm{\beta}}$ is available. Let $\mathbf{V}$ be the $q\times q$ orthogonal matrix whose columns $\mathbf{v}_{i}$ are orthonormal eigenvectors of $\mathbf{X}^{T}\mathbf{X}$, and let $\lambda_{1}\geq\lambda_{2}\geq\dots\geq\lambda_{q}>0$ be the eigenvalues of $\mathbf{X}^{T}\mathbf{X}$. For any fixed $q\times 1$ vector $\mathbf{c}$, there is a unique $q\times 1$ vector $\bm{\alpha}$ whose elements satisfy

[TABLE]

Vector $\mathbf{c}$ defines an effect $\mathbf{c}^{T}\bm{\beta}$. By the Guass-Markov theorem, the minimum-variance unbiased linear estimator for this effect is $\mathbf{c}^{T}\hat{\bm{\beta}}$. Using equation (2), the variance of this estimator $var(\mathbf{c}^{T}\hat{\bm{\beta}})$ can be shown to satisfy

[TABLE]

Under the constraint $\mathbf{c}^{T}\mathbf{c}=1$ which implies $\sum^{p}_{i=1}\alpha^{2}=1$, Silvey (1969) observed from equation (3) that “relatively precise estimation is possible in the directions of latent vectors of $\mathbf{X}^{T}\mathbf{X}$ corresponding to large latent roots; relatively imprecise estimation in these directions corresponding to small latent roots.” While one can use this observation to find $\mathbf{c}$ values such that $\mathbf{c}^{T}\hat{\bm{\beta}}$ have relatively small variances, it does not offer a meaningful interpretation for the underlying effects $\mathbf{c}^{T}{\bm{\beta}}$. Indeed, as Belsley, Kuh and Welsch (2004, p178) had noted that effects defined through the eigenvalues and eigenvectors of $\mathbf{X}^{T}\mathbf{X}$ are unlikely to be of practical interest.

In this paper, we focus on a class of effects of practical interest and look among them for estimable effects. This class consists of all effects involving only $\beta_{1},\beta_{2},\dots,\beta_{p}$, so they have a clear interpretation as group effects for variables in the strongly correlated group. We are interested in finding estimable effects in this class; as none of the underlying parameters is estimable, such estimable effects are of theoretical interest and they also have practical applications some of which are mentioned in our motivations below. Specifically, we study the class of effects, $\Xi^{\prime}$, given by

[TABLE]

where $\mathbf{w^{\prime}}=(w^{\prime}_{1},w^{\prime}_{2},\dots,w^{\prime}_{p})^{T}$ is any $p\times 1$ vector satisfying $\sum_{i=1}^{p}|w^{\prime}_{i}|=1$. We call $\Xi^{\prime}$ the class of normalized group effects of the $p$ strongly correlated variables. We choose constraint $\sum_{i=1}^{p}|w^{\prime}_{i}|=1$ instead of $\sum_{i=1}^{p}(w^{\prime}_{i})^{2}=1$ because it allows $\Xi^{\prime}$ to include commonly used weighted averages such as $\frac{1}{p}\sum^{p}_{i=1}\beta_{i}$, even though it is technically more difficult to handle as it is non-smooth. We call an effect $\xi(\mathbf{w^{\prime}})$ in $\Xi^{\prime}$ a group effect and the corresponding vector $\mathbf{w^{\prime}}$ a weight vector. Individual parameters $\beta_{1},\beta_{2},\dots,\beta_{p}$ are special group effects in $\Xi^{\prime}$ but they are not estimable. Our objective is to look for a characterization of the estimable effects in this class.

We are motivated by the fact that estimable effects are useful for inference and estimation concerning the underlying parameters $\beta_{1},\beta_{2},\dots,\beta_{p}$. They are also useful for knowing when accurate predictions can be made using the estimated model. For example, if the estimated value $\hat{\xi}(\mathbf{w^{\prime}})$ of an estimable effect $\xi(\mathbf{w^{\prime}})$ is significantly different zero, then we may reject the null hypothesis $H_{0}:\hskip 0.72229pt\beta_{1}=\beta_{2}=\dots=\beta_{p}=0$ and conclude that one or more of the variables in this group are not zero. Also, each effect represents a linear constraint on the parameters, so an estimable effect can be used for dimension reduction. The parameter space for the underlying parameters is $\mathbb{R}^{p}$. If $\xi(\mathbf{w^{\prime}})$ is estimable, then the unknown parameters satisfy

[TABLE]

This reduces the parameter space from $\mathbb{R}^{p}$ to essentially a line in $\mathbb{R}^{p}$, and such a dimension reduction can be used for estimating the underlying parameters. The set of weight vectors $\{\mathbf{w}^{\prime}\}$ associated with the estimable effects also defines a region in the space of the $p$ strongly correlated variables over which accurate predictions can be made using the estimated parameters $\hat{\beta}_{1},\hat{\beta}_{2},\dots,\hat{\beta}_{p}$, even though these are not accurate estimates of $\beta_{1},\beta_{2},\dots,\beta_{p}$.

To make the characterization problem manageable, we focus on a uniform model for which the level of multicollinearity generated by strongly correlated variables can be quantified and the impact of multicollinearity on group effects is mathematically tractable. Under this model, we find an optimal effect that benefits from multicollinearity in the sense that the variance of its minimum-variance unbiased linear estimator actually decreases when the level of multicollinearity increases; that is, this effect can be more accurately estimated when the level of multicollinearity goes higher. It is rather surprising that such a linear combination of the parameters exists as these parameters themselves are not estimable at high levels of multicollinearity. This optimal effect is the only one in class $\Xi^{\prime}$ that benefits from multicollinearity; other effects all suffer in that the variances of their minimum-variance unbiased linear estimators all go to infinity as the level of multicollinearity approaches the extreme. The optimal effect has a simple interpretation as a variability weighted average of the underlying parameters. At any given level of multicollinearity, all estimable effects are located around this effect. The uniform model captures effectively the impact of multicollinearity on group effects of strongly correlated variables in (1). We demonstrate through numerical examples that the variability weighted group effect for (1) is also estimable. As an example of its applications, we also discuss a constrained local regression method that uses the variability weighted average effect to estimate the underlying parameters. This method complements the Ridge regression (Horel and Kennard, 1970) and other penalized methods such as Lasso (see, e.g., Hastie, Tibshirani and Wainwright, 2015) in that it is a local method for estimating the parameters of only strongly correlated variables; the least-squares estimates of other parameters are unchanged.

The rest of this paper is organized as follows. In Section 2, we introduce the uniform model under which we reduce $\Xi^{\prime}$ to a subclass $\Xi$. This subclass is only “$(1/2^{p})$th” the size of $\Xi^{\prime}$ but it contains all effects that can be most accurately estimated. We then find the optimal variability weighted average effect through $\Xi$ and give a characterization of all estimable effects under the uniform model using this effect as a reference point. In Section 3, we present a numerical study on the variability weighted group effect and discuss the constrained local regression method. Proofs of theorems and corollaries are given in the Appendix. It should be noted that strong correlations among predictor variables represents only one type of multicollinearity. We focus on this one type because it is the most common, and because it is local in nature so it can be isolated and modelled. The local nature of this type of multicollinearity is also discussed in the Appendix.

2 The optimal and estimable effects in uniform models

2.1 The uniform model

Consider a simple case of (1)

[TABLE]

with $p$ predictors variables in $\mathbf{X}=[\mathbf{x}_{1},\mathbf{x}_{2},\dots,\mathbf{x}_{p}]$ where each column $\mathbf{x}_{i}$ has a mean of $\bar{x}_{i}=0$ and a length of $\|\mathbf{x}_{i}\|=1$, and $\sigma^{2}=1$. In the context of studying multicollinearity, there must be two or more variables in the model, so $p\geq 2$. For this model, $\mathbf{X}^{T}\mathbf{X}$ is the correlation matrix; that is,

[TABLE]

where $r_{ij}=corr(\mathbf{x}_{i},\mathbf{x}_{j})$. We assume that $n>p$ and variables $\mathbf{x}_{i}$ are linearly independent so that the least-squares estimator for $\bm{\beta}$,

[TABLE]

is available. We define a uniform model as a linear model (5) that satisfies the following two conditions:

[TABLE]

The uniform model is an approximation to a linear model with a group of strongly correlated predictor variables. If the group does not satisfy condition (8), we may change the signs of some of the variables so that it does without affecting the underlying model; the impact of this sign change is absorbed by a corresponding sign change of the parameters. Under condition (8), as correlation coefficients of strongly correlated variables, $r_{ij}$ must all be close to but below 1. As such they are also close to each other, which implies condition (9) holds approximately. Theorem 2.1 below gives a sufficient condition for obtaining (8) through sign changes of some variables in the group.

Theorem 2.1

Let $\mathbf{x}_{1},\mathbf{x}_{2},\dots,\mathbf{x}_{p}$ be predictor variables in model (5) such that $\bar{x}_{i}=0$ and $\|\mathbf{x}_{i}\|=1$ for $i=1,2,\dots,p$. Suppose

[TABLE]

Then, among the $2^{p}$ sets of size $p$ each, formed by choosing exactly one element from $\{\mathbf{x}_{j},-\mathbf{x}_{j}\}$ for $j=1,2,\dots,p$, there exists a set from which every pair of variables has a positive correlation.

We call the set identified by Theorem 2.1 an all positive correlations arrangement of the variables or an APC arrangement for short. Any group of variables satisfying condition (10) has an APC arrangement that may be found by first computing $r_{1j}=corr(\mathbf{x}_{1},\mathbf{x}_{j})$ for $j=2,3,\dots,p$ and then changing all $\mathbf{x}_{j}$ where $corr(\mathbf{x}_{1},\mathbf{x}_{j})<0$ into $-\mathbf{x}_{j}$. The resulting APC arrangement is

[TABLE]

Note that condition (10) requires only the existence of one variable whose correlations with other variables are at least $\sqrt{2}/2$ (roughly 0.7) in absolute value. A strongly correlated group of variables behind a multicollinearity problem usually satisfies this condition. Thus, we may assume its variables are in an APC arrangement or equivalently condition (8) holds without loss of generality.

We use the uniform model for the simplicity it brings to the characterization problem. Specifically, with condition (8) we will be able to reduce the complexity of the characterization problem by a factor of $2^{p}$. With condition (9), we can measure the level of multicollinearity by using $r$; the larger $r$ is, the closer it is to 1, the more ill-conditioned $\mathbf{X}^{T}\mathbf{X}$ in (6) and (7) becomes, and thus the higher the level of multicollinearity. This allows us to control the level of multicollinearity easily through a single parameter $r$, which makes it convenient to study the effect of multicollinearity quantitatively. The condition that $\bar{x}_{i}=0$ and $\|\mathbf{x}_{i}\|=1$ will be relaxed in Section 2.3.

2.2 The average group effect and its optimality

Let $\Xi^{\prime}$ be the class of normalized group effects for parameters $\beta_{1},\beta_{2},\dots,\beta_{p}$ in (5). Consider the subclass $\Xi=\{\xi(\mathbf{w})\}$ consisting of all properly weighted averages where

[TABLE]

and $\mathbf{w}=(w_{1},w_{2},\dots,w_{p})^{T}$, $\sum^{p}_{i=1}w_{i}=1$ and $w_{i}\geq 0$. This subclass contains only “$(1/2^{p})$” of the effects in $\Xi^{\prime}$ in that every effect $\xi(\mathbf{w})$ in $\Xi$ corresponds to $2^{p}$ effects $\xi(\mathbf{w}^{\prime})$ in $\Xi^{\prime}$ where each of the $2^{p}$ $\mathbf{w}^{\prime}$ is obtained by changing the signs of some elements of $\mathbf{w}$ to negative. The constraint for this class $\sum^{p}_{i=1}w_{i}=1$ and $w_{i}\geq 0$ is a simple smooth constraint, which makes it easy to work with.

The minimum-variance unbiased linear estimator of $\xi(\mathbf{w})$ is

[TABLE]

where $\hat{\bm{\beta}}=(\hat{\beta}_{1},\hat{\beta}_{2},\dots,\hat{\beta}_{p})^{T}$ is the least-squares estimator of $\bm{\beta}$ in (7). The optimal effect in $\Xi$ is defined as the one whose estimator $\hat{\xi}(\mathbf{w})$ has the smallest variance. The average of the parameters is an effect in $\Xi$ defined by weight vector $\mathbf{w}_{0}=(1/p,1/p,\dots,1/p)^{T}\in\mathbb{R}^{p}.$ We denote this effect by $\tau_{a}$ and call it the average group effect of the variables; that is,

[TABLE]

Correspondingly, let $\mathbf{w}_{j}=(w_{1},w_{2},\dots,w_{p})^{T}$ where $w_{j}=1$ and $w_{k}=0$ for $k\neq j$. We call

[TABLE]

the individual effect of variable $\mathbf{x}_{j}$. We will first show that the average group effect $\tau_{a}$ is the optimal in $\Xi$ and then show that it is also the optimal over the much larger class $\Xi^{\prime}$. To this end, we present Theorem 2.2 below which gives the variance of $\hat{\xi}(\mathbf{w})$.

Theorem 2.2

Suppose conditions (8) and (9) hold. Then, the variance of a minimum-variance unbiased linear estimator $\hat{\xi}(\mathbf{w})$ in (12) is

[TABLE]

The following corollaries highlight the impact of multicollinearity on the variances of estimators for individual and average group effects. To emphasize their dependence on the level of multicollinearity $r$, we will write the variances as $var(\hat{\xi}(\mathbf{w}),r)$. For the estimator of an individual effect $\hat{\beta}_{j}$, we have

Corollary 2.1

Suppose conditions (8) and (9) hold. Then, for a fixed $r$,

[TABLE]

Further, $var(\hat{\beta}_{j},r)$ is a strictly monotone increasing function of $r$ and

[TABLE]

Corollary 2.1 confirms the known result that the variance of the unbiased estimator for $\beta_{j}$ goes to infinity when the level of multicollinearity goes to the extreme. It also shows that among the subclass of effects $\Xi$ defined in (11), individual parameters are the most difficult to estimate in that the variances of their unbiased estimators $\hat{\beta}_{j}=\hat{\xi}(\mathbf{w}_{j})$ are the largest among unbiased estimators (12) for this subclass of effects. This observation is also valid over the larger class $\Xi^{\prime}$. We now give the fixed-$r$ and asymptotic optimality of the average group effect $\tau_{a}$.

Corollary 2.2

Suppose conditions (8) and (9) hold. Then, at any fixed $r$

[TABLE]

Also, $var(\hat{\tau}_{a},r)$ is a strictly monotone decreasing function of $r$ with

[TABLE]

Further, among variances $var(\hat{\xi}(\mathbf{w}),r)$ of unbiased estimators (12) for effects in $\Xi$, $var(\hat{\tau}_{a},r)$ is the only one that remains bounded as $r\rightarrow 1$.

Corollary 2.2 shows the average group effect $\tau_{a}$ is the optimal effect in $\Xi$ as its minimum-variance unbiased estimator $\hat{\xi}(\mathbf{w}_{0})$ has the smallest variance among all estimators (12) for effects in $\Xi$. Its proof also shows at any fixed $r$, other estimable effects ${\xi}(\mathbf{w})$ are all around ${\xi}(\mathbf{w}_{0})$ in that their associated weight vectors $\mathbf{w}$ are in a neighborhood of $\mathbf{w}_{0}$; see the remark after the proof of Corollary 2.2 in the Appendix for a detailed discussion. The proof also shows that for any fixed $\mathbf{w}\neq\mathbf{w}_{0}$, $var(\hat{\xi}(\mathbf{w}),r)\rightarrow\infty$ as $r\rightarrow 1$. Because the minimum value $var(\hat{\xi}(\mathbf{w}_{0}),r)$ is a decreasing function of $r$, the convergence to infinity by $var(\hat{\xi}(\mathbf{w}),r)$ for $\mathbf{w}\neq\mathbf{w}_{0}$ is not uniform. There is in fact a small neighborhood of $\mathbf{w}_{0}$, say $\{\mathbf{w}\hskip 3.61371pt|\hskip 3.61371pt\mathbf{w}\in\mathbb{R}^{p},\hskip 3.61371pt\|\mathbf{w}-\mathbf{w}_{0}\|<\delta(r)\}$, over which $var(\hat{\xi}(\mathbf{w}),r)$ becomes smaller when $r$ increases. But this neighborhood becomes smaller and smaller as its radius $\delta(r)\rightarrow 0$ when $r\rightarrow 1$.

Table 1 contains $var(\hat{\tau}_{a},r)$ and $var(\hat{\beta_{j}},r)$ values at various $r$ levels for a uniform model with $p=8$ variables. There are two points to note. (i) The $r$ value in column 1 starts at $r=0$ which corresponds to an orthogonal design, and increases to $r=0.999$ which represents very strong correlation. In column 2, $var(\hat{\tau}_{a},r)$ decreases from $0.125$ to $0.015$; the latter is about 12% of the former. Thus the average group effect is much more accurately estimated under strong correlations among the variables. (ii) Comparing columns 2 and 3, as $r$ increases we see a clear divergence of the two variances with $var(\hat{\tau}_{a},r)$ going down to its limit of $1/p^{2}=1/8^{2}$ and $var(\hat{\beta_{j}},r)$ going up to infinity. This shows the type of multicollinearity generated by strong correlations among variables represents a redistribution of information in favor of the average group effect $\tau_{a}$ at the expense of individual effects $\beta_{j}$.

We now go back to the class of all normalized group effects $\Xi^{\prime}$ in (4). For an effect $\xi(\mathbf{w^{\prime}})$ in $\Xi^{\prime}$ but not in $\Xi$, one or more of the elements of $\mathbf{w}^{\prime}=(w^{\prime}_{1},w^{\prime}_{2},\dots,w^{\prime}_{p})^{T}$ must be negative. Consider the corresponding effect $\xi(\mathbf{w})$ in $\Xi$ where $\mathbf{w}=(w_{1},w_{2},\dots,w_{p})^{T}$ and $w_{i}=|w_{i}^{\prime}|$ for $i=1,2,\dots,p$. Noting that Theorem 2.2 applies to minimum-variance unbiased estimators (12) for all effects $\xi(\mathbf{w^{\prime}})$ in $\Xi^{\prime}$ under the uniform model and since

[TABLE]

by (14), we have

[TABLE]

In the sense of (19), the subclass $\Xi$ contains all effects of $\Xi^{\prime}$ that can be most accurately estimated, so the average group effect $\tau_{a}$ is also the optimal over the much larger class of effects $\Xi^{\prime}$. For any $\mathbf{w}^{\prime}\neq\mathbf{w}_{0}$, we can also prove that $var(\hat{\xi}(\mathbf{w^{\prime}}),r)$ goes to infinity as $r$ approaches 1.

2.3 The variability weighted group effect for a general uniform model

Consider a more general linear model

[TABLE]

where $\mathbf{y}$ is an $n\times 1$ vector of response variable values, $\beta_{0}$ the unknown intercept term, $\mathbf{1}_{n}$ the $n\times 1$ vector of $1$’s, and $\mathbf{X}=[\mathbf{x}_{1},\mathbf{x}_{2},\dots,\mathbf{x}_{p}]$ the known $n\times p$ design matrix, $\bm{\beta}=(\beta_{1},\beta_{2},\dots,\beta_{p})^{T}$ the unknown vector of parameters of variables $\mathbf{x}_{i}$, and $\bm{\varepsilon}$ the $n\times 1$ vector of random error satisfying

[TABLE]

We assume variables in (20) satisfy $corr(\mathbf{x}_{i},\mathbf{x}_{j})=r>0$ for all $(i,j)$ and call such a model the general uniform model. Comparing with the simple uniform model (5), the general uniform model has an intercept and there are no restrictions on the mean or length of variables $\mathbf{x}_{i}$. Let

[TABLE]

where $x_{ij}$ is the $i$th element of variable $\mathbf{x}_{j}$ and $\bar{x}_{j}$ the mean of its elements for $j=1,2,\dots,p$. Without loss of generality, we assume that $s_{j}>0$ for all $j$. To focus on the impact of the level of multicollinearity $r$ on effects in $\Xi^{\prime}$, we hold $s_{j}$ which also affect the accuracy of estimation fixed, so our results here are conditional on $s_{j}$. Let $\mathbf{w}_{w}=(w_{1}^{*},w_{2}^{*},\dots,w_{p}^{*})^{T}$ where

[TABLE]

We define a variability weighted group effect, $\tau_{w}$, for variables in (20) as

[TABLE]

In (23), the weight $w^{*}_{i}$ for parameter $\beta_{i}$ is large if the variability of $\mathbf{x}_{i}$, represented by $s_{i}$, is large relative to that of other variables. Since parameters for variables with larger variabilities tend to be more accurately estimated, this variability weighted group effect $\tau_{w}$ has the appeal of giving more weight to the more accurately estimated individual effects. More importantly, $\tau_{w}$ corresponds to the average group effect for variables in the simple uniform model (5). This connection gives it the optimality enjoyed by the average group effect in (5). Theorem 2.3 below further illustrates this point.

Theorem 2.3

For model (20), suppose $corr(\mathbf{x}_{i},\mathbf{x}_{j})=r>0$ for all $(i,j)$. Let $\hat{\bm{\beta}}$ be the least-squares estimator of $\bm{\beta}$ and $\hat{\tau}_{w}$ be the minimum-variance unbiased linear estimator for $\tau_{w}$ where

[TABLE]

Then, $var(\hat{\tau}_{w},r)$ is a strictly monotone decreasing function $r$ and

[TABLE]

Further, among all estimators (12) for group effects in $\Xi^{\prime}$ for model (20), $\hat{\tau}_{w}$ is the only one with a bounded variance when $r$ approaches 1.

By Theorem 2.3, $\tau_{w}$ is the optimal effect among the class of normalized group effects $\Xi^{\prime}$ for model (20) in an asymptotic sense as the variance of its estimator $var(\hat{\xi}(\mathbf{w}_{w}),r)$ is asymptotically the smallest as $r$ approaches 1. For a fixed $r$, the variance of an estimator $var(\hat{\xi}(\mathbf{w}),r)$ also depends on $s_{i}$, so $var(\hat{\xi}(\mathbf{w}_{w}),r)$ may not be the smallest. Thus $\tau_{w}$ may not be the exact optimal but it is nearly optimal when $r$ is large. All estimable effects for model (20) are in a neighborhood of $\tau_{w}$. The average group effect $\tau_{a}$ for (20) is estimable when $s_{i}$ do not differ too much. In this case, individual elements of $\mathbf{w}_{w}$ are all roughly $1/p$ and the average group effect is in the neighborhood of $\tau_{w}$. At a fixed $r$, the most difficult effect to estimate is the individual effect $\beta_{j}$ of the variable $\mathbf{x}_{j}$ with the smallest variablility $s_{j}$. See the proof of Theorem 2.3 in the Appendix and the remark after the proof for details.

3 Numerical examples and applications

We have conducted a detailed analysis of the multicollinearity problem in uniform models (5) and (20), which are made of a single group of strongly correlated variables with a uniform correlation structure. For a general model (1) having one or more groups of strongly correlated variables with non-uniform correlation structures as well as not strongly correlated variables, a similarly detailed analysis of the problem seems to be difficult. Fortunately, the multicollinearity problem due to a group of strongly correlated variables is local in nature in that parameters for variables outside of the group are little affected by the problem. Also, as we have noted in the discussion following (8) and (9) that the uniform structure is a good approximation to the correlation structure of a group of strongly correlated variables in its APC arrangement. So the uniform models provide a useful local approximation to the general model (1) in terms of this type of multicollinearity problem.

Indeed, much of what we have learned through the uniform models are still useful for a general model (1). Specifically, (i) the within group APC arrangement is effective for finding estimable effects of a group of strongly correlated variables in model (1), (ii) the variability weighted group effect $\tau_{w}$ of such a group defined under its APC arrangement is estimable, in fact approximately optimal, and it can be substantially more accurately estimated than the average of parameters of the same number of uncorrelated variables, and (iii) $\tau_{w}$ still serves as a reliable location around which other estimable effects may be found. We now illustrate these observations with numerical examples. We also apply $\tau_{w}$ for inference and estimation of the underlying parameters.

3.1 Numerical examples

For economy of space, we use the following setup for all examples in this subsection. Consider a linear model (1) with 10 predictor variables $\mathbf{x}_{1},\mathbf{x}_{2},\dots,\mathbf{x}_{10}$ in three groups $\{\mathbf{x}_{1},\mathbf{x}_{2}\}$, $\{\mathbf{x}_{3},\mathbf{x}_{4},\mathbf{x}_{5}\}$ and $\{\mathbf{x}_{6},\mathbf{x}_{7},\dots,\mathbf{x}_{10}\}$. We use two parameters $w_{1},w_{2}\in[0,1]$ and 10 independent $n$-variate standard normal random vectors $\mathbf{z}_{i}$ to generate different levels of within group correlations as follows:

[TABLE]

So this model (1) contains two separate groups of correlated predictor variables as well as five independent predictor variables. The theoretical non-zero correlation coefficients are:

[TABLE]

These correlation coefficients are all positive, so the two groups of correlated variables come with the APC arrangement. The numerical examples given below involve observed values of variables $\mathbf{x}_{i}$. There are mild correlations among the observed values of the last five variables. The observed correlation coefficients among variables in the two correlated groups also differ somewhat from the theoretical values above, and they do not have a uniform structure.

For all cases of this model (1) that we discuss below, the sample size and the true parameter values are: $n=15$, $\sigma^{2}=1$ and

[TABLE]

For ease of comparison, the 10 variables are conveniently organized in two pairs of groups, $\{\mathbf{x}_{1},\mathbf{x}_{2}\}$ and $\{\mathbf{x}_{6},\mathbf{x}_{7}\}$, and $\{\mathbf{x}_{3},\mathbf{x}_{4},\mathbf{x}_{5}\}$ and $\{\mathbf{x}_{8},\mathbf{x}_{9},\mathbf{x}_{10}\}$. In each pair, both groups have the same number of variables but only the first group contains correlated variables. We compare estimated values and their variances of the following average and variability weighted group effects:

1. $\tau_{1}$: the average group effect for correlated group $\{\mathbf{x}_{1},\mathbf{x}_{2}\}$.

      2. $\tau_{2}$: the average group effect for correlated group $\{\mathbf{x}_{3},\mathbf{x}_{4},\mathbf{x}_{5}\}$.

      3. $\tau_{3}$: the average group effect for independent group $\{\mathbf{x}_{6},\mathbf{x}_{7}\}$.

      4. $\tau_{4}$: the average group effect for independent group $\{\mathbf{x}_{8},\mathbf{x}_{9},\mathbf{x}_{10}\}$.

      5. $\tau_{1}^{w}$: the variability weighted group effect for correlated group $\{\mathbf{x}_{1},\mathbf{x}_{2}\}$.

      6. $\tau_{2}^{w}$: the variability weighted group effect for correlated group $\{\mathbf{x}_{3},\mathbf{x}_{4},\mathbf{x}_{5}\}$.

      7. $\tau_{3}^{w}$: the variability weighted group effect for independent group $\{\mathbf{x}_{6},\mathbf{x}_{7}\}$.

      8. $\tau_{4}^{w}$: the variability weighted group effect for independent group $\{\mathbf{x}_{8},\mathbf{x}_{9},\mathbf{x}_{10}\}$.

Table 2 to Table 6 contain numerical results for five different cases defined by different $(w_{1},w_{2})$ values. Two of these involve adjustments to the predictor variables $\mathbf{x}_{i}$ for illustrating the importance of the APC arrangement and the advantage of $\tau^{w}_{i}$ over $\tau_{i}$. Each table represents one case, and it contains the mean and variance of 1000 estimates (12) of each $\tau$ in the list above. These 1000 estimates are generated as follows. We first generate a design matrix $\mathbf{X}$ through randomly chosen $\mathbf{z}_{i}$ values and (26) for the case. Then, with this same design matrix we generate 1000 $\mathbf{y}$-observations, $\mathbf{y}_{1},\mathbf{y}_{2},\dots,\mathbf{y}_{1000}$, each of length $n=15$, using parameters in (27). The resulting 1000 pairs of $(\mathbf{y}_{i},\mathbf{X})$ are then used to compute 1000 least-squares estimates $\hat{\bm{\beta}}$ and then 1000 estimates (12) for each $\tau$. For example, with a pair of $(\mathbf{y}_{i},\mathbf{X})$ the average group effect $\tau_{2}$ and the weighted group effect $\tau^{w}_{2}$ are estimated by

[TABLE]

where $\hat{\beta}_{j}$ are the least-squares estimates of $\beta_{j}$, and weights $w^{*}_{j}$ are defined by (22) and computed using the observed values of the second group of correlated variables $\{\mathbf{x}_{3},\mathbf{x}_{4},\mathbf{x}_{5}\}$. As a reference, we also include in each table the means and variances of 1000 least-squares estimates for 8 individual effects $\beta_{j}$ (only 8 are included to fit the space). Although the numerical results are tied to the $\mathbf{X}$ used, since this simulation can be easily reproduced and results do not depend on $\mathbf{X}$ very much as long as it is generated randomly through (26), we do not show the $\mathbf{X}$ matrix used.

Case 1 (Table 2) represents a model containing two separate weakly correlated groups. With $(w_{1},w_{2})=(0.3,0.4)$, the observed correlations from the design matrix $\mathbf{X}$ used to compute Table 2 range from 0 to 0.57. So there is no strong correlation or multicollinearity here. The right-side of Table 2 shows the least-squares estimators for individual effects have small biases and variances. The left-side shows all 8 group effects are accurately estimated. Here, because the estimators such as (28) are unbiased, they are accurate when their variances are small. Comparing the variances of estimators for $\tau_{i}$ with that for $\tau^{w}_{i}$, we see the latter are slightly smaller in this case of weakly correlated data.

Case 2 (Table 3) represents a model containing two strongly correlated groups. With $(w_{1},w_{2})=(0.9,0.95)$, the observed correlations among variables within each of the first two groups are all over 0.90. So there is multicollinearity from these two groups/sources. The right-side of Table 3 shows the least-squares estimators for individual effects of the 5 variables in these two groups doing badly with large variances. But that for the 5 (theoretically) uncorrelated individual effects (only 3 are shown) are not affected and their variances are comparable to the corresponding variances in Table 2. The left-side shows all 8 group effects are also accurately estimated with small variances. In particular, variances for weighted group effects of the two correlated groups ($\tau^{w}_{1}$ and $\tau_{2}^{w}$) dropped substantially from Case 1, indicating that increased correlation/multicollinearity is making them more accurately estimated, whereas variances of other effects changed relatively little.

Case 3 (Table 4) represents a model with extreme multicollinearity. With $(w_{1},w_{2})=(0.999,0.999)$, the observed correlations among variables within each of the first two groups are all over 0.999. The right-side of Table 5 shows the least-squares estimators for individual effects of the 5 correlated variables brake down with variances in the order of $10^{5}$. But that for the 5 uncorrelated effects (only 3 are shown) are again not affected, illustrating the local nature of this type of multicolinearity problem. The left-side shows all 8 group effects are also accurately estimated with small variances. In particular, variances for $\tau^{w}_{1}$ and $\tau^{w}_{2}$ are now so small that they are close to the theoretical lower limits (25) for the uniform model (not shown). The small variances associated with the average group effects $\tau_{1}$ and $\tau_{2}$ of the two correlated groups reflect the remark after Theorem 2.3; the $s_{i}$ of the variables here are all close, so the average group effect is an estimable effect. This observation is further confirmed in Case 4 (Table 5) where the level of multicollinearity has been lowered to $(w_{1},w_{2})=(0.99,0.99)$ from Case 3, but variables $\mathbf{x}_{2}$ and $\mathbf{x}_{5}$ are replaced with $2\mathbf{x}_{2}$ and $2\mathbf{x}_{5}$. This change creates a situation where $s_{2}$ dominates $s_{1}$, and $s_{5}$ dominates $s_{3}$ and $s_{4}$. So the average group effects $\tau_{1}$ and $\tau_{2}$ are no longer near the weighted group effects $\tau^{w}_{1}$ and $\tau^{w}_{2}$. Consequently, they are not estimable and their variances become very large whereas that of the two weighted group effects are not affected by this change.

Case 5 (Table 6) illustrates the dependence of the weighted group effect on the (within group) APC arrangement of the variables. For this case, we changed the signs of $\mathbf{x}_{2}$ and $\mathbf{x}_{5}$, so everything else about the model is still the same except variables in the two strongly correlated groups are not all positively correlated. The cancellation phenomenon from the uniform model [see (43)] responsible for the optimality of the average and weighted group effects disappears. Both the average and the weighted group effects have large variances, and become not estimable.

The above examples are a part of a larger numerical study on the variability weighted group effect in a general model (1). The study shows that under the within group APC arrangement, while the estimator of the variability weighted group effect can no longer achieve the limiting variance in Theorem 2.3 for model (20), it is nevertheless remarkably accurate. Its variance is also a decreasing function of the level of multicollinearity. These suggest a perfect uniform correlation structure is only needed for archiving the limiting variance. Without the uniform structure, the weighted group effect is still accurately estimated. The local nature of this type of multicollinearity problem is also clearly shown by the examples; see the Appendix for further discussions.

3.2 Constrained local regression method

Ridge regression of Horel and Kennard (1970) may be used to estimate parameters of strongly correlated variables. However, this method shrinks all estimated values of parameters in the model, whether or not they are that of the strongly correlated variables. This is undesirable as the impact of the multicollinearity problem here is local; least-squares estimates for variables not in the strongly correlated group are unaffected and need not be changed. In the following, we propose a follow-up procedure to the least-squares method which retains the unaffected estimates and deals with the inference and estimation of the affected parameters separately using the weighted group effect.

Suppose there are $p$ predictor variables in a strongly correlated group in its APC arrangement, and denote by $\bm{\beta}_{p}^{t}\in\mathbb{R}^{p}$ the vector of parameters for variables in this particular group. Then, the variability weighted group effect for this group is

[TABLE]

where $\mathbf{w}_{w}\in\mathbb{R}^{p}$ is the known weight vector computed using this group of variables and (22), and $\bm{\beta}_{p}^{t}$ and $\tau_{w}$ are unknown. Since $\tau_{w}$ can be accurately estimated by $\hat{\tau}_{w}$, replacing the unknown $\tau_{w}$ with the estimated $\hat{\tau}_{w}$ and the unknown $\bm{\beta}_{p}^{t}$ with a general $\bm{\beta}_{p}\in\mathbb{R}^{p}$ in equation (29), we obtain a line

[TABLE]

which the true value $\bm{\beta}_{p}^{t}$ should be close to, that is, $\hat{\tau}_{w}\approx\mathbf{w}_{w}^{T}\bm{\beta}_{p}^{t}$. To make use of this information to estimate $\bm{\beta}_{p}^{t}$, we first look for the point $\bm{\beta}^{*}_{p}$ on line (30) with the shortest distance to the origin. This point serves as a “lower bound” for feasible estimates for $\bm{\beta}_{p}^{t}$; we should not shrink the estimate for $\bm{\beta}_{p}^{t}$ so much that its norm is below $\|\bm{\beta}^{*}_{p}\|$. Explicit formula is available for computing $\bm{\beta}^{*}_{p}$ using $\hat{\tau}_{w}$ and $\mathbf{w}_{w}$. Once $\bm{\beta}^{*}_{p}$ is computed, we explore other points on line (30) by setting $\|\bm{\beta}_{p}\|^{2}=c$ where $c$ is a constant larger than $\|\bm{\beta}^{*}_{p}\|^{2}$. The line in (30) intersects the $p$-sphere defined by $\|{\bm{\beta}}_{p}\|^{2}=c$ at two points. Each point represents an estimate for the unknown $\bm{\beta}^{t}_{p}$, and the final selection of $c$ and the point as the estimated value of $\bm{\beta}_{p}^{t}$ can be made through various means such as cross-validation.

Table 7 is an example that illustrates the above method using the model in the last subsection containing two correlated groups of variables. This table is computed as follows. We first generated a design matrix $\mathbf{X}$ using the model with $(w_{1},w_{2})=(0.85,0.8)$. The observed correlations among variables in the first two groups of variables range from 0.85 to 0.98, so there are strong correlations among these variables. We then generated one $\mathbf{y}$ observation, computed the least-squares estimates for individual effects as well as the weighted group effects for the two groups in Table 7.

Table 7 indicates that individuals effects of variables in Groups 1 and 2 are all not significant. For Group 2, this is due to the large variances for these effects caused by the underlying multicollinearity since we know the true values are not zero. At the bottom of the table, we give the weighted group effects for these two groups of correlated variables and their standard errors. These group effects are accurately estimated with small variances. The $t$-test for the first group effect $\tau_{1}^{w}$ has a large $p$-value. This suggests two possibilities: either the two underlying parameters of Group 1 are both zero or they are not zero but they cancelled out after weighting. When the second possibility is considered unlikely, we may conclude the parameters of Group 1 are both zero. We do so in this example, so we drop these two variables from the model which makes it unnecessary to estimate their parameters. The second group effect $\tau^{w}_{2}$ has a very small $p$-value. This indicates the underlying parameters of Group 2 are not all zero, and we proceed to estimate these parameters.

The weight vector for $\tau_{2}^{w}$ is $\mathbf{w}_{w}=(0.3712,0.3218,0.3068)^{T}$, so the corresponding estimated weighted group effect line (30) is

[TABLE]

Plugging the true value of $\bm{\beta}_{p}^{t}=(\beta_{3},\beta_{4},\beta_{5})=(1,2,3)$ into the right-hand side of this equation, we obtain a value of 1.935, so the true value is close to this line by this measure. For line (31),

[TABLE]

which may be viewed as an initial estimate of $\bm{\beta}^{t}_{p}$. The squared minimum distance of line (31) to the origin is $\|\bm{\beta}^{*}_{p}\|^{2}=10.2104$. Exploring other points on the line by setting $c=\|\bm{\beta}_{p}^{*}\|^{2}+3=13.2104$ (this $c$ is chosen for convenience, it is not optimized), one of the two points we obtain is

[TABLE]

which would be a good estimate of $\bm{\beta}_{p}^{t}$. We call the above a constrained local regression as it uses the estimated effect line (31) as a constraint to estimate only the parameters of a strongly correlated group of variables. A detailed study of this method is in progress and results will appear elsewhere.

4 Concluding remarks

We note that for a group of $p$ strongly correlated predictor variables in model (1), the exact optimal effect in $\Xi^{\prime}$, $\tau^{*}={\xi}(\mathbf{w}^{*})$, can be computed numerically. Here, $\Xi^{\prime}$ is the one defined for parameters of this group only. There are a total of $2^{p}$ sign arrangements of these $p$ variables. The APC arrangement is an example of these. We may view $\Xi^{\prime}$ as the union of $2^{p}$ sets $\Xi^{\prime}_{k}$, $k=1,2,\dots,2^{p}$, each defined by one sign arrangement of the variables and all with the same constraint $\sum^{p}_{i=1}w_{i}=1$ and $w_{i}\geq 0$. To find $\tau^{*}$, we compute the optimal effect $\tau^{*}_{j}$ in each $\Xi^{\prime}_{k}$ and $\tau^{*}$ is the $\tau^{*}_{j}$ whose estimator (12) has the smallest variance among all $2^{p}$ such optimal effects. In numerical examples we have examined, $\tau^{*}$ is always an optimal from a $\Xi^{\prime}_{k}$ with an APC arrangement. When the correlations among the $p$ variables are all strong, $\tau^{*}$ is approximately the same as the weighted group effect ${\tau}_{w}={\xi}(\mathbf{w}_{w})$ under the same APC arrangement in that $\mathbf{w}^{*}\approx\mathbf{w}_{w}$ and $var(\hat{\tau}^{*})\approx var(\hat{\tau}_{w})$; in this sense we say that $\tau_{w}$ is approximately optimal. Substantial difference between $\hat{\tau}^{*}$ and $\hat{\tau}_{w}$ is observed when the underlying variables are not strongly correlated, but in this case individual parameters are accurately estimated and group effects such as $\hat{\tau}^{*}$ and $\hat{\tau}_{w}$ are of little interest.

There does not seem to be a meaningful interpretation for the exact optimal effect $\tau^{*}$ beyond that it is the numerical optimal. It is in general difficult to compute. Without a simple analytic expression, it also does not lend itself to a theoretical analysis. In contrast, the weighted group effect $\tau_{w}$ is easy to compute, always accurately estimated and approximately the optimal under the APC arrangement of the underlying variables. Its theoretical properties under the uniform model also add to its appeal. We recommend it for testing the hypothesis of no group effect for a group of strongly correlated predictor variables and for estimating the parameters of such variables.

Finally, we have taken a modelling approach to study the multicolinearity problem instead of the traditional approach which focuses on the detection and remedies for the problem. This has provided a more detailed look at the type of the problem generated by strongly correlated variables and allowed us to characterize estimable effects centered around the weighted group effect. We are studying this type of multicollinearity problem under other commonly used parametric models for the correlation structure, and hope to further develop this modelling based approach.

5 Appendix

5.1 Proof of Theorem 2.1

For brevity, we only sketch a geometric proof below. For any given $i$, since $\bar{x}_{i}=0$ and $\|\mathbf{x}_{i}\|=1$,

[TABLE]

where $\theta_{i}$ is the angle between $\mathbf{x}_{i}$ and $\mathbf{x}_{1}$. It follows from (10) that

[TABLE]

for $i=2,3,\dots,p$. Let $\theta$ be the angle between a vector $\mathbf{x}$ and $\mathbf{x}_{1}$. The region

[TABLE]

is the interior of a double cone with apex at the origin, an apex angle of $\pi/4$ and an axis containing $\mathbf{x}_{1}$. By (32), all $\mathbf{x}_{i}$ are inside this double cone.

Call the cone that contains $\mathbf{x}_{1}$ the first half of the double cone and its opposing cone the second half. If an $\mathbf{x}_{i}$ is in the second half of the double cone, then $-\mathbf{x_{i}}$ must be in the first half. Without loss of generality, suppose $\{\mathbf{x}_{2},\mathbf{x}_{3},\dots,\mathbf{x}_{k}\}$ are in the first half and $\{\mathbf{x}_{k+1},\mathbf{x}_{k+2},\dots,\mathbf{x}_{p}\}$ are in the second half for some $k$ value satisfying $2\leq k\leq p$. Then

[TABLE]

are all in the first half of the double cone. Since the apex angle is only $\pi/4$, the angle $\eta$ between a pair of variables from the set in (33) satisfies $0\leq\eta<\pi/2$. This and the fact that they are all unit vectors imply that their correlation coefficient, which equals $\cos(\eta)$, must be positive. Thus, the set in (33) is one of two such sets where all pairwise correlations are positive. $\Box$

5.2 Proof of Theorem 2.2

Under conditions (8) and (9), matrix $\mathbf{X}^{T}\mathbf{X}$ in (6) is

[TABLE]

where $0<r<1$. The variance of the least-squares estimator $\hat{\bm{\beta}}$ in (7) is

[TABLE]

It follows from (12) and (35) that the variance of $\hat{\xi}(\mathbf{w})$ is

[TABLE]

In order to see clearly the impact of $r$ on $var(\hat{\xi}(\mathbf{w}))$, we now express $(\mathbf{X}^{T}\mathbf{X})^{-1}$ in terms of $r$. By the symmetry imbedded in $\mathbf{X}^{T}\mathbf{X}$ in (34), its inverse should be of a similar form, that is,

[TABLE]

where $t$ and $v$ are unknown constants. Suppose the form in (37) is correct. It follows from $[\mathbf{X}^{T}\mathbf{X}][\mathbf{X}^{T}\mathbf{X}]^{-1}=\mathbf{I}$ that

[TABLE]

Solving the system of equations (38) and (39) for $t$ and $v$, we obtain

[TABLE]

and

[TABLE]

It can be readily verified that the matrix in (37) with elements given by (40) and (41) is indeed the unique inverse of $\mathbf{X}^{T}\mathbf{X}$ in (34). Thus the variance of an estimator given in (36) can be expressed as

[TABLE]

This and (40) and (41) then imply the theorem. $\Box$

5.3 Proofs of Corollaries 2.1 and 2.2

Proof of Corollary 2.1. The weight vector associated with $\hat{\beta}_{j}$ is $\mathbf{w}_{j}=(w_{1},w_{2},\dots,w_{p})^{T}$ where $w_{j}=1$ and $w_{k}=0$ for $k\neq j$. For any fixed $r$, at $\mathbf{w}=\mathbf{w}_{j}$, the $\sum^{p}_{i=1}w_{i}^{2}$ term in the numerator of (14) reaches its maximum value of 1 and the $\sum_{i<j}w_{i}w_{j}$ term reaches its minimum of 0. Further, the denominator in (14) is a positive constant since $0<r<1$. These imply (15).

The above discussion also gives the following simplified version of (14) for $var(\hat{\beta}_{j},r)$,

[TABLE]

It can be verified using the above expression that $var(\hat{\beta}_{j},r)$ is a strictly monotone increasing function of $r$ with the limit showing in (16). $\Box$

Proof of Corollary 2.2. For any $\mathbf{w}=(w_{1},w_{2},\dots,w_{p})^{T}$ satisfying $\sum^{p}_{1=i}w_{i}=1$ and $w_{i}\geq 0$, we have

[TABLE]

Thus, when the first term on the right-hand side $\sum^{p}_{1=i}w_{i}^{2}$ is minimized, the second term $2\sum_{i<j}w_{i}w_{j}$ is maximized. The first term is minimized at $\mathbf{w}=\mathbf{w}_{0}=(1/p,1/p,\dots,1/p)^{T}\in\mathbb{R}^{p}$. So the numerator of (14) is minimized at $\mathbf{w}_{0}$. This proves (17).

At $\mathbf{w}=\mathbf{w}_{0}$, the numerator of (14) is

[TABLE]

The denominator of (14) can be factored as follows,

[TABLE]

It follows from (14), (43) and (44) that

[TABLE]

Equation (45) shows $var(\hat{\tau}_{a},r)$ is a monotone decreasing function of $r$ and it also implies (18).

Finally, let $\mathbf{w}$ be an arbitrary but fixed weight vector such that $\mathbf{w}\neq\mathbf{w}_{0}$. Since function $\sum^{p}_{i=1}w_{i}^{2}$ attains its minimum of $\sum^{p}_{i=1}\frac{1}{p^{2}}$ at only $\mathbf{w}=\mathbf{w}_{0}$, there exists a small constant $\delta>0$ for this $\mathbf{w}$ such that

[TABLE]

This and (42) imply that for this fixed $\mathbf{w}$,

[TABLE]

Thus by (14), the variance for the estimator of the effect defined by this $\mathbf{w}$ is

[TABLE]

The numerator of (46) approaches $p\delta>0$ as $r$ approaches 1, but the denominator approaches 0. Thus, for this fixed $\mathbf{w}$ the variance $var(\hat{\xi}(\mathbf{w}),r)$ goes to infinity as $r$ approaches 1. The last part of the corollary follows from this observation, (18) and the arbitrary selection of $\mathbf{w}\neq\mathbf{w}_{0}$. $\Box$

To see that all estimable effects ${\xi}(\mathbf{w})$ are all around ${\xi}(\mathbf{w}_{0})$ in the sense of their associated weight vectors $\mathbf{w}$ are around $\mathbf{w}_{0}$, by (46) the variance $var(\hat{\xi}(\mathbf{w}),r)$ is a strictly monotone increasing function of $\delta$ which measures the distance between $\mathbf{w}$ and $\mathbf{w}_{0}$. The collection of estimable effects are given by weight vectors $\mathbf{w}$ with $\delta$ values that are small relative to $(1-r)$ and are thus all around ${\xi}(\mathbf{w}_{0})$.

5.4 Proof of Theorem 2.3

Let $\bar{y}$ be the mean of elements of $\mathbf{y}$, and let

[TABLE]

Then, $\mathbf{y}^{\prime}$ is the centered response variable with mean 0, and $\mathbf{x}^{\prime}_{j}$ the standardized predictor variable with mean 0 and length 1. In terms of these variables, model (20) becomes the standardized model

[TABLE]

where $\mathbf{X}^{\prime}=[\mathbf{x}^{\prime}_{1},\mathbf{x}^{\prime}_{2},\dots,\mathbf{x}^{\prime}_{p}]$ is the new $n\times p$ design matrix whose columns are unit vectors with mean zero. The intercept term of (20) is removed when $\mathbf{y}$ is centered but it can be easily recovered using parameters of (47). Since $corr(\mathbf{x}^{\prime}_{i},\mathbf{x}^{\prime}_{j})=corr(\mathbf{x}_{i},\mathbf{x}_{j})=r$ for all $(i,j)$, $\mathbf{X^{\prime}}^{T}\mathbf{X^{\prime}}$ has the uniform structure in (34). We now show that results obtained for model (5) in Section 2.2 apply to the standardized model (47).

Let $\hat{\bm{\beta}}^{\prime}$ be the least squares estimator for $\bm{\beta}^{\prime}$, i.e.,

[TABLE]

Since $\mathbf{X^{\prime}}^{T}\mathbf{X^{\prime}}$ has the uniform structure, from the proof of Theorem 2.2 we see that results for model (5) in Section 2.2 would apply to the standardized model (47) if

[TABLE]

To show this is true, first note that

[TABLE]

where $\mathbbm{1}$ is an $n\times n$ matrix of 1’s. So by (48),

[TABLE]

Next, noting that $Var(y)=\sigma^{2}\mathbf{I}$ and that $\mathbbm{1}_{n\times n}\mathbf{X}^{\prime}=0$ since $\mathbf{X}^{\prime}$ is the matrix of standardized variables with mean zero, we obtain (49). Since $\mathbf{X^{\prime}}^{T}\mathbf{X^{\prime}}$ has the uniform structure, (49) shows that if we add a $\sigma^{2}$ factor to all expressions of variance in theorems and corollaries for model (5) in Section 2.2, then they apply to model (47).

Parameters $\bm{\beta}$ and $\bm{\beta}^{\prime}$ in models (20) and (47) satisfy $\bm{\beta}^{\prime}=\mathbf{S}\bm{\beta}$ where $\mathbf{S}=diag(s_{1},s_{2},\dots,s_{p})$. Their least-squares estimators satisfy

[TABLE]

The minimum-variance unbiased linear estimator for the average group effect of the standardized model (47) is

[TABLE]

Using Corollary 2.2, we can show its variance $var(\hat{\tau}^{\prime}_{a},r)$ is a strictly monotone decreasing function of $r$ with limit

[TABLE]

We now express the variance of an estimator $\hat{\xi}(\mathbf{w})$ in (12) for an effect $\xi(\mathbf{w})$ of the original model (20) in terms of that for the corresponding effect of the standardized model (47). By (50), $\hat{\bm{\beta}}=\mathbf{S}^{-1}\hat{\bm{\beta}}^{\prime}$. So

[TABLE]

where $h(\mathbf{w}^{T}\mathbf{S}^{-1})$ denotes the sum of the elements of $\mathbf{w}^{T}\mathbf{S}^{-1}$. Since the inverse $\mathbf{S}^{-1}=diag(s_{1}^{-1},s_{2}^{-1},\dots,s_{p}^{-1})$, by (22),

[TABLE]

where $\mathbf{1}_{p}$ is the $p\times 1$ vector of 1’s. This and (53) imply that

[TABLE]

It follows from this that

[TABLE]

Thus

[TABLE]

Corollary 2.2 and (52) then imply $var(\hat{\tau}_{w},r)$ is a strictly monotone decreasing function of $r$ and has the limit showing in (25).

Finally, to see that $\hat{\tau}_{w}$ is the only one among estimators for the class $\Xi^{\prime}=\{\xi(\mathbf{w^{\prime}})\}$ that has a bounded variance when $r$ goes to 1, it suffices to consider only estimators for effects in the subclass $\Xi=\{\xi(\mathbf{w})\}$. By (53), any estimator $\hat{\xi}(\mathbf{w})$ in this subclass can be expressed as

[TABLE]

where $\mathbf{w}_{+}=\mathbf{S}^{-1}\mathbf{w}/h(\mathbf{w}^{T}\mathbf{S}^{-1})$ which is a proper weight vector. Thus

[TABLE]

which implies $var(\hat{\xi}(\mathbf{w}))$ is bounded when $r$ approaches 1 if and only if the corresponding $var(\mathbf{w}_{+}^{T}\hat{\bm{\beta}}^{\prime})$ is bounded. By Corollary 2.2, $var(\mathbf{w}_{+}^{T}\hat{\bm{\beta}}^{\prime})$ is bounded if and only if $\mathbf{w}_{+}=\mathbf{w}_{0}$. Since $\mathbf{w}_{+}=\mathbf{w}_{0}$ if and only if $\mathbf{w}=\mathbf{w}_{w}$, it follows that $var(\hat{\xi}(\mathbf{w}_{w}))$ is the only variance among $var(\hat{\xi}(\mathbf{w}))$ for estimators of effects in $\Xi^{\prime}$ that remains bounded when $r$ approaches 1. $\Box$

For a fixed $r$, the variance of an estimator $var(\hat{\xi}(\mathbf{w}),r)$ in (56) also depends on $s_{i}$, so $var(\hat{\xi}(\mathbf{w}_{w}),r)$ may not be the smallest and consequently $\tau_{w}$ may not be the exact optimal. But all estimable effects for model (20) are in a neighborhood of $\tau_{w}$. To see this, we have noted after the proof of Corollary 2.2 that for the simple uniform model (5), estimable effects are in a neighborhood of the average group effect $\tau_{a}$. Thus, estimable effects for model (47) are in a neighborhood of $\tau^{\prime}_{a}$. By (55), an effect for model (20) is a linearly transformed effect for model (47). Transforming the estimable effects of model (47) using (55), we obtain estimable effects for model (20) which form a neighborhood around $\tau_{w}$.

At a fixed $r$, the most difficult effect to estimate is the individual effect $\beta_{j}$ of variable $\mathbf{x}_{j}$ with the smallest variablility $s_{j}$. To see this, by (56), the minimum-variance unbiased linear estimator for a $\beta_{i}$ has variance

[TABLE]

Under the standardized model (47), by Corollary 2.1 and its proof, the $var(\hat{\beta_{i}}^{\prime})$ are the same for $i=1,2,\dots,p$ and they are the largest among variances of estimators for effects of model (47). This observation, (56) and (57) then imply that $var(\hat{\beta}_{j})=var(\hat{\xi}(\mathbf{w}_{j}))$ is the largest among all $var(\hat{\xi}(\mathbf{w}))$ for estimators (12) of effects in $\Xi^{\prime}$.

5.5 The local nature of the multicollinearity problem

We have seen through numerical examples that the multicollinearity problem generated by a group of strongly correlated variables is a local problem in that its impact is limited to least-squares estimates of parameters for variables in the group. This was a motivating factor behind our approach of isolating and modelling such a multicollinearity problem with the uniform models. If the problem is not local, then local models such as the uniform models would be neither justified nor helpful for understanding the impact of the problem. Here, we provide further discussion on the local nature of the problem.

Consider a simple case of model (1) where $\sigma^{2}=1$, $p=3$ and

[TABLE]

so only $\mathbf{x}_{1}$ and $\mathbf{x}_{2}$ are strongly correlated and variable $\mathbf{x}_{3}$ is outside of the strongly correlated group. The inverse of $\mathbf{X}^{T}\mathbf{X}$ is

[TABLE]

Variances of the estimated individual effects for the strongly correlated $\mathbf{x}_{1}$ and $\mathbf{x}_{2}$ are over 9.0, whereas that for $\mathbf{x}_{3}$ is less than 2.0. Unusually large values in $(\mathbf{X}^{T}\mathbf{X})^{-1}$ are responsible for this substantial difference, and they appear only in elements $(i,j)$ of $(\mathbf{X}^{T}\mathbf{X})^{-1}$ where both $i$ and $j$ belong to the two strongly correlated variables. This always occurs when multicollinearity is caused by strongly correlated variables. To see this, consider a modified (5) with $\bar{x}_{i}=0$ and $\|\mathbf{x}_{i}\|=1$ but only $k$ ($2\leq k<p$) of its variables are strongly correlated. The $k\times k$ block of $\mathbf{X}^{T}\mathbf{X}$ representing the correlation matrix of these $k$ variables contains elements whose values are all close to 1. They dominate other off-diagonal elements of $\mathbf{X}^{T}\mathbf{X}$ which are small and random correlation coefficients involving other variables. As such, they make the rows/columns they reside in nearly linearly dependent; see (58) for an example. Since $\mathbf{X}^{T}\mathbf{X}$ has one or two more such nearly linearly dependent column/rows than its submatrices for the minors of elements in the $k\times k$ block, it is “more singular” than these submatrices. Because of this, the determinant of $\mathbf{X}^{T}\mathbf{X}$ is small in absolute value relative to the cofactors associated with these minors. By Cramer’s rule, large values must appear in the corresponding $k\times k$ block in $(\mathbf{X}^{T}\mathbf{X})^{-1}$, making individual effects of strongly correlated variables difficult to estimate as their least-squares estimators’ variances are the diagonal elements of this block. But this problem does not affect other individual effects (e.g., that of $\mathbf{x}_{3}$) as their associated submatrices contain the same number of nearly linearly dependent row/columns as $\mathbf{X}^{T}\mathbf{X}$. This makes their associated cofactors equally small as the determinant of $\mathbf{X}^{T}\mathbf{X}$, so their least-squares estimators’ variances are not large.

The above discussion shows that this type of multicollinearity problem is a local problem in that it affects only the strongly correlated variables even when there are other variables in the model.