Minimizing Oracle-Structured Composite Functions

TL;DR

This paper introduces a versatile optimization method for minimizing composite convex functions that leverages both gradient evaluations and structured problem solutions, demonstrating improved efficiency in data-intensive applications.

Contribution

The authors develop a novel optimization algorithm that combines quasi-Newton, bundle methods, and damping, requiring minimal assumptions and parameter tuning, applicable to diverse problem types.

Findings

Outperforms standard solvers on data-heavy problems

Effective across stochastic, utility, and risk-averse optimization tasks

Does not require tuning of algorithm parameters

Abstract

We consider the problem of minimizing a composite convex function with two different access methods: an oracle, for which we can evaluate the value and gradient, and a structured function, which we access only by solving a convex optimization problem. We are motivated by two associated technological developments. For the oracle, systems like PyTorch or TensorFlow can automatically and efficiently compute gradients, given a computation graph description. For the structured function, systems like CVXPY accept a high level domain specific language description of the problem, and automatically translate it to a standard form for efficient solution. We develop a method that makes minimal assumptions about the two functions, does not require the tuning of algorithm parameters, and works well in practice across a variety of problems. Our algorithm combines a number of well-known ideas, including a low-rank quasi-Newton approximation of curvature, piecewise affine lower bounds from bundle-type methods, and two types of damping to ensure stability. We illustrate the method on stochastic optimization, utility maximization, and risk-averse programming problems, showing that our method is more efficient than standard solvers when the oracle function contains much data.