Linear Regression with Limited Observation

TL;DR

This paper introduces efficient algorithms for linear regression variants that operate under limited attribute observations, achieving comparable or better accuracy with fewer attributes, and resolves an open problem in the field.

Contribution

The paper presents novel algorithms for Ridge, Lasso, and Support-vector regression with limited attribute access, improving attribute efficiency and solving an open problem.

Findings

Algorithms match full-information attribute requirements for Ridge and Lasso.

Support-vector regression requires exponentially fewer attributes than previous methods.

Experimental results confirm theoretical advantages and superior performance.

Abstract

We consider the most common variants of linear regression, including Ridge, Lasso and Support-vector regression, in a setting where the learner is allowed to observe only a fixed number of attributes of each example at training time. We present simple and efficient algorithms for these problems: for Lasso and Ridge regression they need the same total number of attributes (up to constants) as do full-information algorithms, for reaching a certain accuracy. For Support-vector regression, we require exponentially less attributes compared to the state of the art. By that, we resolve an open problem recently posed by Cesa-Bianchi et al. (2010). Experiments show the theoretical bounds to be justified by superior performance compared to the state of the art.