Tactics for Improving Least Squares Estimation

TL;DR

This paper introduces tactics like MM principle, Moreau envelopes, and proximal distance methods to enhance the speed and efficiency of least squares regression in high-dimensional settings, validated through numerical experiments.

Contribution

It presents novel computational strategies for fast least squares estimation, including the use of surrogate functions, smoothing techniques, and distance penalties, with practical Julia implementations.

Findings

Deweighting and Moreau envelope approximations significantly speed up computations.

Numerical experiments demonstrate the effectiveness of proposed tactics.

Julia software implementing these methods is publicly available.

Abstract

This paper deals with tactics for fast computation in least squares regression in high dimensions. These tactics include: (a) the majorization-minimization (MM) principle, (b) smoothing by Moreau envelopes, and (c) the proximal distance principle for constrained estimation. In iteratively reweighted least squares, the MM principle can create a surrogate function that trades case weights for adjusted responses. Reduction to ordinary least squares then permits the reuse of the Gram matrix and its Cholesky decomposition across iterations. This tactic is pertinent to estimation in L2E regression and generalized linear models. For problems such as quantile regression, non-smooth terms of an objective function can be replaced by their Moreau envelope approximations and majorized by spherical quadratics. Finally, penalized regression with distance-to-set penalties also benefits from this perspective. Our numerical experiments validate the speed and utility of deweighting and Moreau envelope approximations. Julia software implementing these experiments is available on our web page.