Penalty, Shrinkage, and Preliminary Test Estimators under Full Model Hypothesis

TL;DR

This paper compares various penalty and preliminary test estimators in multiple regression models, showing that ridge regression consistently outperforms traditional estimators, while other penalty methods outperform least squares in certain conditions.

Contribution

It provides a comprehensive analytical and simulation-based comparison of penalty estimators and traditional estimators under the full model hypothesis in regression.

Findings

Ridge regression uniformly dominates least squares and other estimators.

LASSO, adaptive LASSO, SCAD, and elastic net outperform least squares.

No penalty estimator uniformly dominates Stein-type estimators.

Abstract

This paper considers a multiple regression model and compares, under full model hypothesis, analytically as well as by simulation, the performance characteristics of some popular penalty estimators such as ridge regression, LASSO, adaptive LASSO, SCAD, and elastic net versus Least Squares Estimator, restricted estimator, preliminary test estimator, and Stein-type estimators when the dimension of the parameter space is smaller than the sample space dimension. We find that RR uniformly dominates LSE, RE, PTE, SE and PRSE while LASSO, aLASSO, SCAD, and EN uniformly dominates LSE only. Further, it is observed that neither penalty estimators nor Stein-type estimator dominate one another.