Efficient Learning of Generalized Linear and Single Index Models with Isotonic Regression

TL;DR

This paper introduces efficient algorithms for learning generalized linear and single index models using isotonic regression, improving computational and statistical performance with practical empirical validation.

Contribution

The paper presents novel algorithms for GLMs and SIMs that are both computationally and statistically efficient, overcoming previous limitations requiring fresh samples each iteration.

Findings

Algorithms are both computationally and statistically efficient.

Empirical results demonstrate practical feasibility.

Improved performance over previous methods.

Abstract

Generalized Linear Models (GLMs) and Single Index Models (SIMs) provide powerful generalizations of linear regression, where the target variable is assumed to be a (possibly unknown) 1-dimensional function of a linear predictor. In general, these problems entail non-convex estimation procedures, and, in practice, iterative local search heuristics are often used. Kalai and Sastry (2009) recently provided the first provably efficient method for learning SIMs and GLMs, under the assumptions that the data are in fact generated under a GLM and under certain monotonicity and Lipschitz constraints. However, to obtain provable performance, the method requires a fresh sample every iteration. In this paper, we provide algorithms for learning GLMs and SIMs, which are both computationally and statistically efficient. We also provide an empirical study, demonstrating their feasibility in practice.