A Comparison of First-order Algorithms for Machine Learning

TL;DR

This paper provides a comprehensive comparison of first-order optimization algorithms for convex machine learning problems, highlighting the advantages of primal-dual methods in terms of ease, speed, and accuracy.

Contribution

It offers the first extensive empirical evaluation of various first-order algorithms on both smooth and non-smooth convex problems in machine learning.

Findings

Primal-dual algorithms outperform others in speed and accuracy.

Ease of construction favors primal-dual methods.

Primal-dual algorithms are effective for both smooth and non-smooth problems.

Abstract

Using an optimization algorithm to solve a machine learning problem is one of mainstreams in the field of science. In this work, we demonstrate a comprehensive comparison of some state-of-the-art first-order optimization algorithms for convex optimization problems in machine learning. We concentrate on several smooth and non-smooth machine learning problems with a loss function plus a regularizer. The overall experimental results show the superiority of primal-dual algorithms in solving a machine learning problem from the perspectives of the ease to construct, running time and accuracy.