Gradient Descent Converges Linearly for Logistic Regression on Separable Data

TL;DR

This paper proves that gradient descent with variable learning rates achieves linear convergence for logistic regression on separable data, improving understanding of convergence behavior without strong convexity.

Contribution

It demonstrates that variable learning rates enable linear convergence of gradient descent for logistic regression on separable data, challenging previous assumptions.

Findings

Loss converges exponentially with iterations.

Variable learning rates are crucial for linear convergence.

Sparse logistic regression benefits from improved sparsity-error tradeoff.

Abstract

We show that running gradient descent with variable learning rate guarantees loss $f(x) \leq 1.1 \cdot f(x^*) + \epsilon$ for the logistic regression objective, where the error $\epsilon$ decays exponentially with the number of iterations and polynomially with the magnitude of the entries of an arbitrary fixed solution $x^*$. This is in contrast to the common intuition that the absence of strong convexity precludes linear convergence of first-order methods, and highlights the importance of variable learning rates for gradient descent. We also apply our ideas to sparse logistic regression, where they lead to an exponential improvement of the sparsity-error tradeoff.