# Quickly Finding the Best Linear Model in High Dimensions

## TL;DR

This paper introduces a projected gradient descent algorithm for efficiently finding the optimal linear model in high-dimensional settings, with theoretical guarantees and practical validation.

## Contribution

It presents a novel PGD method with convergence and error bounds applicable to heavy-tailed distributions, without assuming realizability, and includes bias learning augmentation.

## Key findings

- Linear convergence rate established for PGD.
- Effective in heavy-tailed sub-exponential distributions.
- Numerical experiments confirm theoretical predictions.

## Abstract

We study the problem of finding the best linear model that can minimize least-squares loss given a data-set. While this problem is trivial in the low dimensional regime, it becomes more interesting in high dimensions where the population minimizer is assumed to lie on a manifold such as sparse vectors. We propose projected gradient descent (PGD) algorithm to estimate the population minimizer in the finite sample regime. We establish linear convergence rate and data dependent estimation error bounds for PGD. Our contributions include: 1) The results are established for heavier tailed sub-exponential distributions besides sub-gaussian. 2) We directly analyze the empirical risk minimization and do not require a realizable model that connects input data and labels. 3) Our PGD algorithm is augmented to learn the bias terms which boosts the performance. The numerical experiments validate our theoretical results.