Optimal Black-Box Reductions Between Optimization Objectives

TL;DR

This paper introduces optimal and practical black-box reductions that enable the application of algorithms across various optimization settings in machine learning, leading to faster training times and successful practical outcomes.

Contribution

It presents the first set of optimal, practical reductions that unify and extend algorithm applicability across different smoothness and convexity conditions.

Findings

Faster training times for linear classifiers using new reductions

Successful practical experiments demonstrating effectiveness

Unified approach applicable to multiple optimization settings

Abstract

The diverse world of machine learning applications has given rise to a plethora of algorithms and optimization methods, finely tuned to the specific regression or classification task at hand. We reduce the complexity of algorithm design for machine learning by reductions: we develop reductions that take a method developed for one setting and apply it to the entire spectrum of smoothness and strong-convexity in applications. Furthermore, unlike existing results, our new reductions are OPTIMAL and more PRACTICAL. We show how these new reductions give rise to new and faster running times on training linear classifiers for various families of loss functions, and conclude with experiments showing their successes also in practice.