Relative Efficiency of Higher Normed Estimators Over the Least Squares Estimator

TL;DR

This paper compares the efficiency of higher normed estimators, specifically $L_{2k}$, with the traditional least squares estimator, providing conditions, decision rules, and empirical validation for their relative performance.

Contribution

It introduces a testable condition and decision rule for choosing between $L_{2k}$ and $L_2$ estimators, with a focus on the $L_4$ estimator and empirical validation.

Findings

$L_{2k}$ estimator can be more efficient under certain conditions.

A simple decision rule effectively guides estimator choice.

Empirical results confirm the superiority of $L_{2k}$ in real data.

Abstract

In this article, we study the performance of the estimator that minimizes $L_{2k}- $ order loss function (for $ k \ge \; 2 )$ against the estimators which minimizes the $L_2-$ order loss function (or the least squares estimator). Commonly occurring examples illustrate the differences in efficiency between $L_{2k}$ and $L_2 -$ based estimators. We derive an empirically testable condition under which the $L_{2k}$ estimator is more efficient than the least squares estimator. We construct a simple decision rule to choose between $L_{2k}$ and $L_2$ estimator. Special emphasis is provided to study $L_{4}$ estimator. A detailed simulation study verifies the effectiveness of this decision rule. Also, the superiority of the $L_{2k}$ estimator is demonstrated in a real life data set.

Figures

Tables

Table 1. Table 1: Simulated Risk

Sample size	Mixture of T-distribution	(5, -5)	(4,- 4)	(3, -3)	(2, -2)
100	Mixture of T-distn (df=6)	9940	9834	9200	5896
200		9952	9838	9256	5571
500		9961	9844	9309	4861
1000		9965	9856	9337	4309
2000		9975	9896	9348	3763
5000		9973	9912	9445	2833
100	Mixture of T-distn (df=10)	10000	9986	9860	8092
200		9999	9992	9900	8329
500		10000	9997	9959	8842
1000		10000	9996	9973	9041
2000		10000	10000	9991	9368
5000		10000	9999	9988	9666
100	Mixture of T-distn (df=20)	10000	9999	9981	9155
200		10000	10000	9996	9585
500		10000	10000	10000	9846
1000		10000	10000	10000	9950
2000		10000	10000	10000	9984
5000		10000	10000	10000	9999
	Mixture of Beta-distn (asym)	(4,10; 10,4)	(1,4; 4,1)	(2,4; 4,2)	(3,4; 4,3)
100		10000	10000	9691	3632
200		10000	10000	9992	4791
500		10000	10000	10000	7277
1000		10000	10000	10000	9241
2000		10000	10000	10000	9957
5000		10000	10000	10000	10000
	Mixture of Beta-distn (sym)	(4,4; 4,4)	(3,3; 3,3)	(2,2; 2,2)	(1,1; 1,1)
100		2007	3740	7665	9987
200		1881	4748	9445	10000
500		2089	7178	9993	10000
1000		2574	9200	10000	10000
2000		3464	9957	10000	10000
5000		5810	10000	10000	10000
	Mixture of two normal distributions	(3, -3)	(2, -2)	(1, -1)	(0, 0)
100		9999	9806	1862	211
200		10000	9959	1330	26
500		10000	10000	815	0
1000		10000	10000	478	0
2000		10000	10000	265	0
5000		10000	10000	63	0
	Mixture of three normal distributions	(4, -4)	(3, -3)	(2, -2)	(1, -1)
100		9968	9560	5589	550
200		10000	9951	6589	181
500		10000	10000	8252	17
1000		10000	10000	9450	1
2000		10000	10000	9922	0
5000		10000	10000	10000	0

Equations

$$Y=X\beta +\epsilon ,$$

$$\sqrt{n}\left({\hat{\beta }}_{OLS}-{\beta }_0\right)\stackrel{d}{\to }N\left(0,{\sigma }^2(X^{\prime }X{)}^{-1}\right).$$

$$Y=\alpha +\beta x_i+{\epsilon }_i,$$

$$\sqrt{n}\left({\hat{\theta }}_{OLS}-{\theta }_0\right)\stackrel{d}{\to }N\left(0,{\sigma }^2{S}^{-1}\right),$$

$$S=\left(\begin{array}[]{cc}n&\sum_{i=1}^{n}x_{i}\\ \sum_{i=1}^{n}x_{i}&\sum_{i=1}^{n}x^{2}_{i}\end{array}\right).$$

$$\sqrt{n}\left({\hat{\theta }}_4-{\theta }_0\right)\stackrel{d}{\to }N\left(0,\frac{{\mu }_6-{\mu }_3^2}{9{\mu }_2^3}{S}^{-1}\right).$$

$$\sqrt{n}\left({\hat{\theta }}_{L_4}-{\theta }_0\right)\stackrel{d}{\to }N\left(0,V({\theta }_0)\right),$$

$$V({\theta }_0)=A({\theta }_0{)}^{-1}B({\theta }_0)[A({\theta }_0{)}^{-1}{]}^{\prime },$$

$$B({\theta }_0)$$

$$A({\theta }_0)$$

$$\psi=\frac{\delta S_{4}}{\delta\theta}=(-4)\left(\begin{array}[]{c}\sum(Y_{i}-\alpha-\beta x_{i})^{3}\\ \sum(Y_{i}-\alpha-\beta x_{i})^{3}x_{i}\end{array}\right).$$

$$E[\psi(y,\theta_{0})]=(-4)\left(\begin{array}[]{c}n\mu_{3}\\ \mu_{3}\sum x_{i}\end{array}\right),\ \ \ \mbox{and hence},\ \ \ E[\psi(y,\theta_{0})]E[\psi(y,\theta_{0})]^{\prime}=16\mu_{3}^{2}R$$

$$R=\left(\begin{array}[]{cc}n^{2}&n\sum x_{i}\\ n\sum x_{i}&(\sum x_{i})^{2}\\ \end{array}\right).$$

$$Q=\left(\begin{array}[]{cc}n(n-1)&(n-1)\sum_{i=1}^{n}x_{i}\\ (n-1)\sum_{i=1}^{n}x_{i}&(\sum_{i=1}^{n}x_{i})^{2}-\sum_{i=1}^{n}x^{2}_{i}\end{array}\right)$$

$$Q-R$$

$$B({\theta }_0)=16S({\mu }_6-{\mu }_3^2)=16SVar({\epsilon }^3).$$

$$A({\theta }_0)=12{\mu }_2.S$$

$$V({\theta }_0)=A({\theta }_0{)}^{-1}B({\theta }_0)[A({\theta }_0{)}^{-1}{]}^{\prime }=\frac{1}{12{\mu }_2}{S}^{-1}(16S({\mu }_6-{\mu }_3^2)))\frac{1}{12{\mu }_2}{S}^{-1}=\frac{{\mu }_6-{\mu }_3^2}{9{\mu }_2^2}{S}^{-1}.$$

$$\frac{{\mu }_6-{\mu }_3^2}{{\mu }_2^3}<9$$

$$\frac{{\mu }_6}{9{\mu }_2^3}<1.$$

$$X==\left(\begin{array}[]{ccccc}1&x_{11}&x_{21}&\cdots&x_{k1}\\ 1&x_{12}&x_{22}&\cdots&x_{k2}\\ \vdots&\vdots&\vdots&\vdots&\vdots\\ 1&x_{1n}&x_{2n}&\cdots&x_{kn}\\ \end{array}\right).$$

$$Q=\left(\begin{array}[]{ccccc}n(n-1)&(n-1)\sum_{i=1}^{n}x_{1i}&(n-1)\sum_{i=1}^{n}x_{2i}&\cdots&n\sum_{i=1}^{n}x_{ki}\\ (n-1)\sum_{i=1}^{n}x_{1i}&\sum_{i\neq j}x_{1i}x_{1j}&\sum_{i\neq j}x_{1i}x_{2j}&\cdots&\sum_{i\neq j}x_{1i}x_{kj}\\ (n-1)\sum_{i=1}^{n}x_{2i}&\sum_{i\neq j}x_{2i}x_{1j}&\sum_{i\neq j}x_{2i}x_{2j}&\cdots&\sum_{i\neq j}x_{2i}x_{kj}\\ \vdots&\vdots&\vdots&\vdots&\vdots\\ (n-1)\sum_{i=1}^{n}x_{ki}&\sum_{i\neq j}x_{ki}x_{1j}&\sum_{i\neq j}x_{ki}x_{2j}&\cdots&\sum_{i\neq j}x_{ki}x_{kj}\\ \end{array}\right),$$

$$R==\left(\begin{array}[]{ccccc}n^{2}&n\sum_{i=1}^{n}x_{1i}&n\sum_{i=1}^{n}x_{2i}&\cdots&n\sum_{i=1}^{n}x_{ki}\\ n\sum_{i=1}^{n}x_{1i}&(\sum_{i=1}^{n}x_{1i})^{2}&(\sum_{i=1}^{n}x_{1i})(\sum_{i=1}^{n}x_{2i})&\cdots&(\sum_{i\neq j}x_{1i})(\sum_{i=1}^{n}x_{ki})\\ n\sum_{i=1}^{n}x_{2i}&(\sum_{i=1}^{n}x_{2i})(\sum_{i=1}^{n}x_{1i})x_{1j}&(\sum_{i=1}^{n}x_{2i})^{2}&\cdots&(\sum_{i=1}^{n}x_{2i})(\sum_{i=1}^{n}x_{2i}x_{ki})\\ \vdots&\vdots&\vdots&\vdots&\vdots\\ n\sum_{i=1}^{n}x_{ki}&(\sum_{i=1}^{n}x_{ki})(\sum_{i=1}^{n}x_{1i})&(\sum_{i=1}^{n}x_{ki})(\sum_{i=1}^{n}x_{2i})&\cdots&(\sum_{i=1}^{n}x_{ki})^{2}\\ \end{array}\right).$$

$$\frac{{\mu }_6-{\mu }_3^2}{9{\mu }_2^3}<1.$$

$$\frac{Var({\epsilon }^{2k-1})}{(2k-1{)}^2(Var(\epsilon ){)}^{2k-1}}<1.$$

$$\frac{Var({\epsilon }^{2k-1})(2k-3{)}^2}{(2k-1{)}^2(Var({\epsilon }^{2k-3})){\mu }_2^2}<1.$$

$$f(x)=d{x}^{2k};-c\le x\le c,\text{mbox}wherek\text{mbox}isapositiveinteger.$$

$$\frac{{\mu }_6}{9{\mu }_2^3}=\frac{(2k+3{)}^3}{9(2k+1{)}^2(2k+7)}.$$

$${\mu }_r=\frac{a^{r+1}-(-a{)}^{r+1}}{(r+1)\{a-(-a)\}},$$

$$\frac{{\mu }_6}{9{\mu }_2^3}=\frac{3}{7}<1.$$

$${\mu }_{2r}={\sigma }^{2r}(2r-1)\times (2r-3)\times \cdots \times 5\times 3\times 1,$$

Full text

Key Words: Least Squares, Higher Order Loss Function.

**JEL Classification: C01, C13 **

1 Introduction

The least squares (LS hereafter) method is possibly the most popular method of estimation routinely used to estimate the underlying (regression) parameters. Stigler (1981) rightly said: ”The method of least squares is the automobile of modern statistical analysis: Despite its limitations, occasional accidents, and incidental pollution, it and its numerous variations, extensions, and related conveyances carry the bulk of statistical analysis, and are known and valued by nearly all”. Such an overwhelming popularity of the LS may be due to its simplicity, optimal properties and robustness to any distributional assumption. Moreover, it leads to the best (minimum variance) estimator under normality. Laplace used the name ”most advantageous method”. However, it appears to us that such an irresistible popularity of the LS may have impeded the exploration of other smooth loss functions. Comparative computational difficulty might be another reason that such exploration was not favoured by pioneers such as Gauss, Laplace and others. Whereas, a large literature to incorporate non-smooth loss functions in order to address (outlier) robustness have been developed. Unfortunately and surprisingly, whole statistical literature is somewhat mute on the possible use of smooth higher order loss functions. Therefore, it is a pertinent question to ask: Are there any relative advantages in using higher order smooth loss function compared to the omnipresent least squares? Our aim here is to study an appropriate higher order estimator and compare its efficiency against the LS. In the regression set up, we find a significantly large and useful class of error distributions for which a higher order loss function is more efficient than the LS. In this paper, we give an empirically testable condition under which a higher order smooth loss functions lead to a more efficient estimator than the LS. We also provide a simple but pragmatic decision rule to make a choice between $L_{2k}$ and $L_{2}$. A detailed simulation study shows the effectiveness of such a decision rule.

In Section 2, we describe the model and develop the methodology needed to compare the efficiency of different loss functions. Section 3 provides various classes of error distributions which are used for comparison of estimator. In Section 4, we provide a decision rule along with its asymptotic properties. Section 5 provides an epilogue where we consider very general classes of parametric distributions on finite support to illustrate the enormous scope of applicability of the $L_{4}-$ based loss functions. Section 6 summarizes the results of simulation study on mixture distributions. Section 7 gives an application to real life data. Section 8 ends with some concluding remarks and identifies possible future directions of research.

2 Model and assumptions

Consider a linear regression set up

[TABLE]

where $Y$ is an $n\times 1$ vector of observations, $X$ is an $n\times k$ design matrix and $F$ is the cumulative distribution function (cdf) corresponding to the error vector the error $\varepsilon.$ We also assume the following regular conditions:

$\mbox{plim}\left(\frac{X^{\prime}X}{n}\right)=A$, a finite and nonsingular matrix. 2. 2.

$E(\varepsilon|X)=0$. 3. 3.

$E(\varepsilon\varepsilon^{\prime}|X)=\sigma^{2}I$. 4. 4.

Observations are independent.

In this set up, the ordinary least squares estimate (OLS) ${\widehat{\beta}}_{OLS}$ is the best linear unbiased estimator in the sense of minimum variance. It is well-known that for the OLS estimator, under $\beta=\beta_{0}$,

[TABLE]

Furthermore, it is clear that the minimization of $\sum_{i=1}^{n}(Y_{i}-X_{i}^{\prime}\beta)^{k}$ is pointless if $k$ is odd; and the minimization of $\sum_{i=1}^{n}|Y_{i}-X_{i}^{\prime}\beta|$ or $\sum_{i=1}^{n}|Y_{i}-X_{i}^{\prime}\beta|^{k}$ for odd $k$ will not be very convenient because of lack of differentiability $wrt\;\;\beta$. Therefore, it remains to check whether minimization of $\sum_{i=1}^{n}(Y_{i}-X_{i}^{\prime}\beta)^{2k},$ for some positive integer $k$ other than 1, can yield a better result than ${\widehat{\beta}}_{OLS}$, at least in some cases. If so, our objective is to identify those cases. It is obvious that the corresponding estimators for such a cases will be non-linear. Furthermore, when the error is normal, then the best linear unbiased estimator is indeed the best unbiased estimator. Thus, for normal or near normal error, ${\widehat{\beta}}_{OLS}$ always will be better than any other estimator. A closer look reveals that deviation from uni-modality causes the robustness properties of LS to falter.

Studying the efficacy of higher order normed based estimator is important on its own right; not necessarily in comparison to least square. It opens up the possibility to consider a convex combination of loss functions of various degrees. Arthanari and Dodge (1981) considers convex combination of $L_{1}$ and $L_{2}$ norms; and studies its properties. Convex combination of $L_{1},\ldots,L_{p}$ may lead to more useful estimator; and the resultant estimator is expected to be robust to any distributional assumption. Earlier also, such an use of higher order loss functions is attempted. Turner (1960) heuristically touch upon the possible use of a higher order loss function in the context of estimation of the location parameter. He discusses several kinds of general PDFs; and advices in the case of the double exponential, to minimize the sum of the absolute deviations; in the case of the normal, to minimize the sum of the squared deviations (least squares); and in the case of the q-th power distribution, to minimize the sum of the q-th power of the deviations (least q-th’s). Attempts are also made to define a general class of likelihood to derive a robust parameter estimates, robust to distributional assumption. For example, Zeckhauser and Thompson (1970) defines a general class of distribution; and empirically found its suitability.

2.1 Methodology

For the exposition purpose, let us first consider the simple bivariate linear regression model

[TABLE]

for $i=1,\ldots,n$. The usual approach to take the error function as $S_{2}=\sum^{n}_{i=1}(Y_{i}-\alpha-\beta x_{i})^{2}$. We obtain ${\widehat{\theta}}_{OLS}=({\widehat{\alpha}}_{OLS}~{}~{}{\widehat{\beta}}_{OLS})^{\prime}$ by minimizing $S_{2}$ with respect to $\alpha$ and $\beta$. Note that ${\widehat{\theta}}_{OLS}$ is the best estimates in the class of linear unbiased estimators. Hence there may be some nonlinear estimator with better efficiency.

In contrast, we shall take $S_{4}=\sum^{n}_{i=1}(Y_{i}-\alpha-\beta x_{i})^{4}$ as our loss function, and derive ${\widehat{\theta}}_{L_{4}}=\left({\widehat{\alpha}}_{L_{4}}~{}~{}{\widehat{\beta}}_{L_{4}}\right)$ as the estimator of $\theta$. Our objective is to compare ${\widehat{\theta}}_{OLS}$ and ${\widehat{\theta}}_{L_{4}}$, and discover conditions under which the latter performs better than the former. Both of these estimators are $M$-estimators. So they possess the properties such as consistency and asymptotic normality under some standard conditions.

For the OLS estimator,

[TABLE]

where

[TABLE]

We exhibit that ${\widehat{\theta}}_{L_{4}}$ satisfies the following result.

Lemma 1.

[TABLE]

Proof: For the $L_{4}$ estimator, using the M-estimator property, we have

[TABLE]

where

[TABLE]

[TABLE]

and

[TABLE]

Let $\mu_{k}$ denote the $k$th order central moment corresponding to the distribution of $\varepsilon$.

Consequently, we get

[TABLE]

where

[TABLE]

Also, $E[\psi(y,\theta_{0})\psi(y,\theta_{0})^{\prime}]=16(\mu_{6}S+\mu_{3}^{2}Q)$, where

[TABLE]

Thus, $B(\theta_{0})=16\mu_{6}S+16\mu_{3}^{2}(Q-R)$. Since

[TABLE]

[TABLE]

Similarly, one can simplify

[TABLE]

Then

[TABLE]

Hence the proof.

Theorem 1. The $L_{4}$ estimator performs better than the OLS estimator in terms of precision iff

[TABLE]

Proof. Proof follows by comparing 2.2 and 2.3.

For symmetric distribution of $\varepsilon$, or whenever $\mu_{3}=0$, the criterion will be

[TABLE]

Clearly this condition may or may not be satisfied depending on the distribution of $\varepsilon$.

So far, for exposition purpose, we have dealt with a simple regression framework. Now the scope of this paper demands to present all these above findings in a more general regression set-up. The following remark is made to this end.

Remark 1.

For a multiple linear regression model with $k$ regressors, all the above calculations can be carried out with $S=X^{\prime}X$ where

[TABLE]

Whereas $Q$ matrix would be given by

[TABLE]

and the matrix $R$ is given by,

[TABLE]

Thus,  $Q-R=-S$  which gives the earlier result that is, $L_{4}$ estimators are better than that of $L_{2}$ iff

[TABLE]

It may be interesting to examine the performance of $L_{2k}$ relative to that of $L_{2}$, or that of $L_{2k-2}.$ The following two corollaries are presented to this end.

Corollary 1. $L_{2k}$ estimator better than LS (i.e., $L_{2}$) iff

[TABLE]

Proof. The proof is analogous to that of Theorem 1.

Corollary 2. $L_{2k}$ estimator better than $L_{2k-2}$ iff

[TABLE]

Proof. The proof is analogous to that of Theorem 1.

3 OLS versus $L_{4}$ for some selected distributions

In this section, we consider few important parametric error distributions to illustrate the vast scope of applicability of $L_{4}$ based loss function. The list of distributions considered is no way exhaustive, but certainly shows the immense opportunity of applications in diverse areas. Now, we check the aforementioned condition (2.4) hold for different distributions of $\varepsilon$.

3.0.1 U-Shaped Distribution

Consider a simple U-shaped distribution:

[TABLE]

Note that $d=\frac{2k+1}{2c^{2k+1}}.$ It is easy to calculate

[TABLE]

Note that for $k=1,\;\frac{\mu_{6}}{9\mu_{2}^{3}}=\frac{125}{729},$ and in the limit, $\frac{\mu_{6}}{9\mu_{2}^{3}}=\frac{1}{9}.$

A U-shaped distribution has two modes; and can be looked upon as a mixture of two (J-shaped) distributions - a mixture of an extreme positively skewed and another extreme negatively skewed distributions. One popular applied example of a U-shaped distribution is the number of deaths at various ages. Several more examples can be found in by B. S. Everitt ( 2005).

3.0.2 Uniform($-a,a$)

This is a symmetric distribution. Here

[TABLE]

and consequently

[TABLE]

Hence, $L_{4}$ estimator is better than the OLS estimator, when the error component has uniform distribution.

3.0.3 Normal($\mu,\sigma^{2}$)

This is again a symmetric distribution where

[TABLE]

and hence

[TABLE]

Hence, for normally distributed errors, the OLS estimator is always preferred over the $L_{4}$ estimator.

3.0.4 Laplace($\lambda$)

Here we have

[TABLE]

if $r$ is even, and, consequently,

[TABLE]

Hence, when $\varepsilon$ follows Laplace distribution, the OLS estimator is preferred over the $L_{4}$ estimator.

3.0.5 Beta($a,b$)

The beta distribution is a family of continuous probability distributions defined on the interval [0, 1] parametrized by two positive shape parameters, denoted by $a$ and $b$, that appear as exponents of the random variable and control the shape of the distribution. This class of distributions include a variety of symmetric, bell-shaped, positively skewed, negatively skewed, uniform, and ’U-shaped’ distributions. The general form of the central moments of the beta distribution are quite complicated. So we will start with the raw moments and obtain the forms of $\mu_{6}$, $\mu_{3}$ and $\mu_{2}$. Note that, here

[TABLE]

The figure 4 depicted in Section 5, provides a huge range of parameters for which $L_{4}$ is better than $L_{2}.$

3.0.6 Gaussian mixture distribution

Suppose that $F(x)=\frac{1}{2}N(\xi_{1},\sigma^{2})+\frac{1}{2}N(\xi_{2},\sigma^{2})$. We assume, for simplicity, common $\sigma^{2}$ for both the components. Here

[TABLE]

where $\xi=\frac{1}{2}(\xi_{1}+\xi_{2})$ and $\mu_{ji}$ is the $i$th central moment of $N(\xi_{j},\sigma^{2})$ distribution, $j=1,2$. Let $\xi_{2}-\xi=\frac{1}{2}(\xi_{2}-\xi_{1})=-(\xi_{1}-\xi)=a_{0}$. Then (2.7) reduces to

[TABLE]

where $c=\frac{\xi_{2}-\xi_{1}}{2\sigma}$. The left hand side expression of (3.8), which is a function of $c$, is plotted in Figure 1. From the plot we can see for $|c|>1.058$, this function assumes values less than zero and then it decreases rapidly. This means that the condition (2.7) of superiority of $L_{4}$ estimators will be satisfied if the means of the two component of the mixture distribution are more than $1.058\sigma$ distance. A mixture of more than two Gaussian distributions will behave similarly with respect to this condition. Also the case of unequal $\sigma^{2}$ can be tackled similarly.

3.0.7 Truncated Normal distribution

Consider (for simplicity) the both side truncated standard normal distribution. The even order moments are

[TABLE]

Define $\Delta=\sqrt{2\pi}(\Phi(c)-0.5).$ Therefore,

[TABLE]

Now it is easy to calculate $\mu_{6}^{c}=15-\frac{c^{5}exp(-\frac{c^{2}}{2})}{\Delta}-\frac{5c^{3}exp(-\frac{c^{2}}{2})}{\Delta}-\frac{15c\;exp(-\frac{c^{2}}{2})}{\Delta},$ and

$\mu_{4}^{c}=3-\frac{c^{3}exp(-\frac{c^{2}}{2})}{\Delta}-\frac{3c\;exp(-\frac{c^{2}}{2})}{\Delta},$ and

$\mu_{2}^{c}=1-\frac{c\;exp(-\frac{c^{2}}{2})}{\Delta}.$ Now one can see that as long as $c\in(0,2.33),\;L_{4}$ performs better than $L_{2}.$ One implication of this result is that 97 percent times $L_{4}$ performs better than $L_{2}.$

3.0.8 Raised cosine distribution

If $X$ follows a raised cosine distribution with parameters $a$ and $b$, denoted by $X\sim COR(a,b)$, then the probability density function (pdf) is given by

[TABLE]

The form of this distribution resembles that of a normal distribution except for the fact that it has finite tails. Suppose it can be assumed that the value of systematic errors lies in some known interval; and manufacturer has aimed to make device as accurate as possible. In such circumstances, Raised Cosine distribution may be appropriate. Another popular application is in circular data. See Rinne (2010, pp. 116). Other properties like the cdf, moment generating function (mgf), characteristic functions, raw moments up to order 4, and the kurtosis are available in Rinne (2010, pp. 116-118). It is observed that the distribution has a kurtosis of 2.1938, less than that of normal distribution. It has a thin tail. Here, using the mgf, we have

[TABLE]

and hence

[TABLE]

Hence, for this distribution, (2.7) is satisfied, and consequently $L_{4}$ is preferred for parameter estimation.

3.1 A Sub-Gaussian family of distributions

Sub-Gaussian family of distributions is a well-studied family of distribution whose tail is dominated by the normal distribution. As we observed that the $L_{4}$ estimators are preferred for a distribution for which the tail is thinner than that of a normal distribution, here we discuss about a relatively uncommon distribution and validity of the condition with respect to this distribution. Consider a distribution with pdf of the form

[TABLE]

where $k$ is an integer and $c$ is the normalizing constant, which gives

[TABLE]

For various values of $k$, the pdf of the distribution is drawn in Figure 3. Note that $k=1$ provides the normal curve. As $k$ becomes larger and larger, tail of the distribution tend to collapse. For extremely large $k$, the distribution resembles a symmetric curve in a finite support. It is interesting to consider the peaks of all drawn curves. The first plot (the density plot) of this panel shows that $L_{4}$ performs better than $L_{2}$ for all those curves for which peaks are below the red curve. Here it may be mentioned that the red curve is drawn for $k=1.45.$ For the second plot of the panel, various values of $k$ are given in the $x-$axis; and values of the test statistic are given in the $y-$axis. The parallel line, parallel to $x-$axis, shows the cut-off point, which is 1 . The second plot of the panel shows that when the value of $k$ is greater than 1.45, $L_{4}$ performs better than $L_{2}$.

Here $\mu_{r}=0$ if r is odd. When r is even, we have

[TABLE]

We immediately get

[TABLE]

The values of the test statistic against various values of $K$ is drawn in the bottom part in Figure 3. We observe that for $k\geqslant 2$, $\frac{\mu_{6}}{9\mu_{2}^{3}}$ assumes values less than 1.

Now, according to Zeckhauser and Thompson (1970); Turner (1960) and Box and Tiao (1962 ), for a distribution with pdf

[TABLE]

where $k(\sigma,m)=\left[2\sigma\Gamma\left(1+\frac{1}{m}\right)\right]^{-1}$, $L_{m}$ estimators dominate all $L_{m^{\prime}}$ estimators where $m^{\prime}<m$.

Note that, for $m=4$ and $\sigma=1$, we obtain the distribution displayed in Figure 3. So $L_{4}$ estimator performs better than the corresponding $L_{2}$ estimator, that is the OLS estimator, which is entirely in agreement to what we derived above. For the normal distribution $m=2$; $m=1$ gives the double exponential distribution; where $m$ tends to $\infty$, the distribution tends to the rectangular. The article of Zeckhauser and Thompson (1970) considers four empirical examples to find that there is a sizable gains in likelihood if $m$ is estimated rather than pre-specified equal to 2. All of the evidence they found leads them to the conclusion that if accurate estimation of a linear regression line is important, it will usually be desirable to estimate not only the coefficients of the regression line, but also the parameters of the power distribution that generated the errors about the regression line. The effect on the estimates of regression coefficients may not be small.

In the next two section we will construct a decision rule based on the condition (2.6) and carry out some simulation study.

4 Decision rule: OLS versus $L_{4}$

In this Section we derive a decision rule based on the criterion from Section 2 to decide whether OLS or $L_{4}$ estimator is preferred for some data.

Lemma 2. Suppose $X$ follows a distribution for which $\mu_{r}$ exists for all $r$. Then

[TABLE]

Proof: Observe that

[TABLE]

Now all the terms other than the first and second term of (4.9) are of the order $o_{p}(1)$ because $\sqrt{n}(\mu-\overline{x})$ is $O_{p}(1)$, and hence $\sqrt{n}(\mu-\overline{x})^{k},~{}k\geq 2$, is $o_{p}(1)$. Also $\frac{1}{n}\sum_{i=1}^{n}(x_{i}-\mu)^{r-1}=\mu_{r-1}+o_{p}(1)$. Therefore

[TABLE]

$\Box$

Furthermore, by delta method,

[TABLE]

Then, we have the following Theorem.

Theorem 2. Let $v=\frac{\mu_{6}-\mu_{3}^{2}}{\sigma^{6}}$. Suppose $\mu_{12}$ exists for distribution of $X$. Then

[TABLE]

where $\alpha_{0}=\left(1,~{}~{}-(6\mu_{5}-3\mu_{2}\mu_{3}),~{}~{}-\mu_{3},~{}~{}-3\sigma^{4}v\right)$ and Z_{i}=\left(\begin{array}[]{c}(x_{i}-\mu)^{6}-\mu_{6}\\ (x_{i}-\mu)\\ (x_{i}-\mu)^{3}-\mu_{3}\\ (x_{i}-\mu)^{2}-\sigma^{2}\end{array}\right).

Proof of this theorem is given in the Appendix B.

5 An Epilogue

In this section we consider a very general class of parametric distributions on finite support (this assumption is made to ease plot drawing) to illustrate the enormous scope of applicability of $L_{4}$ based loss function. Consider the class of distribution:

[TABLE]

Here $d$ depends on $a$ to make the $f$ a density. The first plot of the panel depicts the shape of the density for different values of $a$. Depending on the value of $a$, this class of distributions includes various ’U-shaped’ (for $a>0$) and ’Bell-Shaped’ (for $a<0$) distributions. It is interesting to consider the peaks of all drawn curves. The first plot (the density plot) of this panel shows that $L_{4}$ performs better than $L_{2}$ for all those curves for which peaks are below the red curve. Here it may be mentioned that the red curve is drawn for $a=-3.2.$ For the second plot of the panel, various values of $a$ are given in the $x-$axis; and values of the test statistic are given in the $y-$axis. The second plot of the panel shows that when the value of $a$ is greater than -3.2, $L_{4}$ performs better than $L_{2}$, in all shape going from low deep to hump till it reaches certain level.

Plots of another parametric family of distributions (belongs to the Pearsonian family, Type II), given by

[TABLE]

are shown below. It may be noted that this particular distribution is linked to $Beta$ distribution as well. To see this, let $Y\sim Beta(\alpha_{1},\alpha_{2}),Y\in(0,1).$ Let $X=a+(b-a)Y.$ Then

[TABLE]

To see the equivalence, set $a=-b=-1,$ and $\alpha_{1}=\alpha_{2}=\alpha+1.$

Figure 5 also depicts the same feature as in Figure 4. Depending on the value of $a$, this group of parametric of distributions depicts various ’U-shaped’ and ’Bell-Shaped’. It is interesting to consider the peaks of all drawn curves. The first plot (the density plot) of this panel shows that $L_{4}$ performs better than $L_{2}$ for all those curves for which peaks are below the red curve. The red curve is drawn for $a=3.5.$ The second plot of the panel shows that when the value of $a$ is greater than 3.5, $L_{4}$ performs better than $L_{2}$, in all shape going from low deep to hump till it reaches certain level.

These two distributions illustrate the enormous possibility of use of $L_{4}$ based loss function. Future studies will investigate whether this is a general phenomenon for other Pearsonian family of errors distributions.

6 Simulation study

We carry out the decision making procedure under 0-1 loss function and calculate the risk function, which is the expected loss. Here we generate data from three types of distribution, one for which $L_{4}$ is always better than OLS estimator, one where OLS is better than $L_{4}$, and the third one is near the boundary. The values of the calculated risk are given in Table LABEL:table_1.

Table 1 given in Appendix A should be here

Simulation study is based on 10000 iterations; and with sample sizes of 100, 200, 500, 1000, 2000, 5000.

The first panel of Table 1 is based on mixture of two $T$ distributions with 6 degrees of freedom (DF) each. Mean of each components are set at $(5,-5),(4,-4),(3,-3),(2,-2).$ Here it may be mentioned that our test needs existence of 6th order moments. To this end, we need $t$ distribution with at least 7 df. DF 6 is considered to examine the performance of our decision rule even when moments do not exist. Mixture coefficients are taken from $U(0,1)$ distribution. From this part of the table, it is clear that decision is more certain as sample size increases; more importantly, it is so when the distance between the two components are more.

The second panel of the Table is based on mixture of two $T$ distribution with 10 df each. Here findings corroborate with the first panel. The third panel is based on mixture of two $T$ distribution with 20 df each.

The 4th panel of the table is based on mixture of two asymmetric $Beta$ distributions. Mixture coefficients are taken from $U(0,1)$ distribution. The fifth panel is based on two symmetric beta distributions with weight from $U(0,1)$. The first column, in Panel 5 needs special attention. The parameter combination $((4,4;4,4))$ is chosen such that it is in the neighborhood of the boundary the test statistic. Here it shows that test does not favour (for the large sample case, n=5000) any one, as expected. Here the risk is near 50% .

The 6th and 7th panel of the Table are based on mixture of two normal distributions. Here also findings are on the expected line. As sample size increases, test correctly discriminates between $L_{4}$ and $L_{2}.$

7 Empirical Illustration

In this sub-section, we provide two illustrations. One is based on a constructed data set which resembles many real life scenario; and the second one is based on a real life data set.

7.1 Constructed Example

Data often contains rounding errors. Variables (like heights or weights, age in years, or birth weight in ounces.) that by their very nature are continuous are, nevertheless, typically measured in a discrete manner. People feel more comfortable to report their age as mid forty, mid fifty and so on. They are rounded to a certain level of accuracy, often to some preassigned decimal point of a measuring scale (e.g., to multiples of 10 cm, 1 cm, or 0.1 cm) or simply our preference of some numbers over other numbers. The reason may be the avoidance of costs associated with a fine measurement or the imprecise nature of the measuring instrument. The German military, for example, measures the height of recruits to the nearest 1 cm. Even if precise measurements are available, they are sometimes recorded in a coarsened way in order to preserve confidentiality or to compress the data into an easy to grasp frequency table.

Here we consider the linear regression where the dependent variable is rounded to nearest integer; independent variables are free of any such errors. The dependent variable is generated as

[TABLE]

However, assume that we do not observe $Y_{st}$ but observe

[TABLE]

Now we are regressing $Y$ on $X_{1}$ and $X_{2}.$ For this example, we set a moderate sample size of 40. We consider 5000 replication.The output is summarized as follows:

Table 2: Average Estimates Based on the Constructed Data.

[TABLE]

It is observed that 90 percent times $L_{4}$ is preferred over $L_{2}$ based on our proposed decision rule.

After estimation of the model, it may be of interest to know which set of estimators provides the best fit. In the present context it is a tricky problem to find an appropriate ’goodness of fit’ measure. Likelihood based methods are not tenable. Similarly, residual sum of square or $R^{2}$ are not not useful to compare the performances of these two set of parameter estimates. Here we suggest to apply the idea of Pseudo $R^{2}$ (See Cameron and Trivedi, 2005 for details, page No. 311).

Let $Q(\theta)$ denotes the objective function being maximized, $Q_{0}$ denotes its value in the intercept-only model, $Q_{fit}$ denotes the value in the fitted model, and $Q_{max}$ denotes the largest possible value of $Q(\theta)$. Then the maximum potential gain in the objective function resulting from inclusion of regressors is $Q_{max}-Q_{0}$ and the actual gain is $Q_{fit}-Q_{0}$. This suggests the measure

[TABLE]

where the subscript RG means relative gain. Note that, for least squares, $R^{2}=R_{RG}^{2}.$ For both the loss functions, $Q_{max}=0.$

We also calculated the number of times the Pseudo $R^{2}$ for $L_{4}$ is numerically greater than that of $L_{2}$. It is astonishing to see that 100 percent times the Pseudo $R^{2}$ for $L_{4}$ is numerically greater than that of $L_{2}$.

7.2 Real Life Example

For our empirical analysis, we use the data provided by the National Sample Survey Organization of India viz. the NSSO 68th round all India unit level survey on consumption expenditure (Schedule1.0, Type 1 and 2) conducted during July 2011 to June 2012. This dataset is a nationally representative sample of household and individual characteristics based on a stratified sampling of households. For this round, the dataset is comprised of 1,68,880 household level observations. The dataset provides a detailed list of various household and individual specific characteristics along with the consumption expenditures of the households. In addition to this, data is also provided on the households’ localization which includes the sector (Rural or Urban), the district and the state/union territory (henceforth, the union territories will be referred to as states). For our analysis, we use the amount of land possessed (in logarithm form) by the households as our principal (dependent) variable along side various demographic variables as controls (independent variables). The kernel density plot clearly suggest that amount of land possession by rural households does have a bimodal distribution 222The same phenomenon is also seen for all-India households (rural and urban together). The plot presented here is for rural household excluding the households with no land. It is interesting to note that bi-modality is observed both for (1) households with non-zero amount of land ; and (2) with all households. All the results presented here are based on rural household with non-zero lands. Number of rural households with non-zero amount of land is 98483. Whole study is based on This set of 98483 observations.. The plot clearly indicates that India is suffering from ”vanishing middle-class syndrome,” only marginal and rich farmers are there. The empirical analysis demands some routine and rudimentary summary statistics as given in Table 2. We regress the amount of land possessed ($Y$) on six explanatory variables, 333 We also tried with many other explanatory variables available in our master file. We also repeated the same exercise for all-india (rural and urban together) households. It is need less to mention that overall findings are same across all models we attempted. viz, Median age of a household (mage), the number of children below 15 years of age (chlt15), the number of old people above 60 years of age (Ogt60), the number of male member in the households (male), the number of female member in the households (female); and finally the number of member with education level above 10th standard (highedu). We then estimate 444In this paper we do not pursue the endogeneity issue, if any. the linear regression model based on $L_{2}$ and $L_{4}.$ The estimated results are summarized as below:

Table 3: Summary Statistics.

[TABLE]

Table 4: Model Estimates and Standard Errors .

[TABLE]

It can be noted that Standard errors (SE) of the parameter estimates are provided in the parenthesis. It is to observe, as expected, that SE of $L_{4}$ based estimates are significantly and uniformly less than that of $L_{2}$ based estimates. The value of our proposed test statistic is 5.30311871 which lies beyond 95 per cent confidence interval (8.90603838 9.09396162) suggesting that $L_{4}$ based estimates are more efficient than that of $L_{2}$ . The pseudo $R^{2}$ for $L_{4}$ is 0.14779372 and the same for $L_{2}$ is 0.09794736 . The pseudo $R^{2}$ also clearly suggests the supremacy of $L_{4}$ over $L_{2}.$

8 Discussion

This paper tried to give answer to the unassailable question: Does higher order loss function based estimator perform better than the omnipresent least squares? Every teacher, student faces this question on the first-day class on regression analysis. We tried to show that, in several real life situations, smooth higher order loss function based estimator may lead to more efficient estimator as compared to universal least squares. It is true that least squares has one unassailable advantages, its simplicity. It may also be computationally less intensive. However, with the advent of modern computing power, computational issues may hardly be relevant.

Further work may commence in the following directions. A generalized version of the condition similar to the one derived in section 4 may be useful for comparing $L_{2k}$ and $L_{2k+2}$ estimators. This may be obtained following a similar approach i.e. by obtaining the variance of these two estimators using the expression of variance of m-estimators and comparing them. However, estimation of higher moments may have impact on the performance of the proposed decision rule. It may be useful to study the impact of outliers on the parameter estimates coming from higher order based loss function. Comparison of break-down point of $L_{2},L_{4}$ estimators may be very useful. It may also be interesting to find robust standard errors for $L_{4}$ for non $i.i.d$ set up.

It may be extremely useful to consider a convex combination of loss functions of various degrees. Arthanari and Dodge (1981) considers convex combination of $L_{1}$ and $L_{2}$ norms; and studies its properties. Convex combination of $L_{1},\ldots,L_{p}$ may lead to more useful estimator; and the resultant estimator is expected to be robust to any distributional assumption. Such combination may give answer to the omnipresent question: What is the optimal loss function for a given data set? The choice and design of loss functions is important in any practical application (see Hennig and Kutlukaya, 2007). Future research will shed light in this direction.

Appendix A

Appendix B

Proof of Theorem 2: We write

[TABLE]

and hence

[TABLE]

$\Box$

Now,

[TABLE]

where $\Gamma=\lim_{n\rightarrow\infty}nE(\overline{Z}\overline{Z}^{\prime})$ and $\overline{Z}=\frac{1}{n}\sum_{i=1}^{n}Z_{i}$. Here

[TABLE]

with

[TABLE]

and

[TABLE]

and hence

[TABLE]

Consequently, we get

[TABLE]

where

[TABLE]

Hence our decision theoretic problem reduces to

[TABLE]

We use the statistic

[TABLE]

which is asymptotically $N(0,1)$, under $A_{0}$.

9 Reference

Arthanari, T. S. and Dodge, Y. (1981), “Mathematical Programming in Statistics,” New York: John Wiley 2. 2.

Everitt B. S. (2005) U-Shaped Distribution, Encyclopedia of Biostatistics, 8, John Wiley & Sons. 3. 3.

Box, G. E. P., and G. C. Tiao (1962): A Further Look at Robustness via Bayes’s Theorem, *Biometrika *, 49, 419-432. 4. 4.

Cameron A. C. and P. Trivedi (2005): Microeconometrics, Cambridge. 5. 5.

Hennig. C, and M. Kutlukaya( 2007): Some Thoughts about the Design of Loss Functions, REVSTAT – Statistical Journal 5, 1, 19–39. 6. 6.

Rinne, H. (2010). Location-Scale Distributions – Linear Estimation and Probability Plotting Using MATLAB. Preprint. 7. 7.

Stigler. S., (1981): Gauss and the Invention of Least Square, The Annals of Statistics, 9,3, 465-474. 8. 8.

Turner, M. C. (1960): On Heuristic Estimation Methods, Biometrics , 16, 299-301. 9. 9.

Zeckhauser, R., and M. Thompson (1970): Linear Regression with Non-Normal Error Terms. The Review of Economics and Statistics, Vol. 52, No. 3, 280-286