New explanations and inference for least angle regression

TL;DR

This paper develops a new theoretical framework for understanding and performing inference with least angle regression (LAR), clarifying its properties, providing a stopping rule, and proposing a modified bootstrap method for uncertainty quantification.

Contribution

It introduces a novel inference framework for LAR, including new mathematical properties, a stopping rule, and a modified bootstrap for better uncertainty assessment.

Findings

LAR estimates of non-zero correlations are normally distributed.

Zero correlations have a non-normal joint distribution.

A modified bootstrap improves inference for LAR.

Abstract

Efron et al. (2004) introduced least angle regression (LAR) as an algorithm for linear predictions, intended as an alternative to forward selection with connections to penalized regression. However, LAR has remained somewhat of a "black box," where some basic behavioral properties of LAR output are not well understood, including an appropriate termination point for the algorithm. We provide a novel framework for inference with LAR, which also allows LAR to be understood from new perspectives with several newly developed mathematical properties. The LAR algorithm at a data level can viewed as estimating a population counterpart "path" that organizes a response mean along regressor variables which are ordered according to a decreasing series of population "correlation" parameters; such parameters are shown to have meaningful interpretations for explaining variable contributions whereby zero correlations denote unimportant variables. In the output of LAR, estimates of all non-zero population correlations turn out to have independent normal distributions for use in inference, while estimates of zero-valued population correlations have a certain non-normal joint distribution. These properties help to provide a formal rule for stopping the LAR algorithm. While the standard bootstrap for regression can fail for LAR, a modified bootstrap provides a practical and formally justified tool for interpreting the entrance of variables and quantifying uncertainty in estimation. The LAR inference method is studied through simulation and illustrated with data examples.