Penalized regression with multiple loss functions and selection by vote

TL;DR

This paper introduces a novel penalized regression method that combines multiple loss functions with a voting procedure for variable selection, enhancing accuracy and computational efficiency in high-dimensional linear models.

Contribution

It proposes a new approach that uses multiple loss functions and a voting scheme for variable selection and estimation, with proven asymptotic efficiency and practical advantages.

Findings

Reduces false discovery rate in variable selection

Improves parameter estimation quality

Enhances computational efficiency

Abstract

This article considers a linear model in a high dimensional data scenario. We propose a process which uses multiple loss functions both to select relevant predictors and to estimate parameters, and study its asymptotic properties. Variable selection is conducted by a procedure called "vote", which aggregates results from penalized loss functions. Using multiple objective functions separately simplifies algorithms and allows parallel computing, which is convenient and fast. As a special example we consider a quantile regression model, which optimally combines multiple quantile levels. We show that the resulting estimators for the parameter vector are asymptotically efficient. Simulations and a data application confirm the three main advantages of our approach: (a) reducing the false discovery rate of variable selection; (b) improving the quality of parameter estimation; (c) increasing the efficiency of computation.