A Novel Gradient Methodology with Economical Objective Function Evaluations for Data Science Applications

TL;DR

This paper introduces a new gradient optimization methodology that efficiently uses objective function evaluations to prevent divergence and explosion issues, improving performance on data science problems.

Contribution

It develops a generic, problem-driven gradient methodology with a novel step size selection, tailored for data science optimization challenges.

Findings

Competitive performance on CUTEst test problems

Superior results on generalized estimating equation model learning

Prevents optimality gap divergence and evaluation explosions

Abstract

Gradient methods are experiencing a growth in methodological and theoretical developments owing to the challenges posed by optimization problems arising in data science. However, such gradient methods face diverging optimality gaps or exploding objective evaluations when applied to optimization problems with realistic properties for data science applications. In this work, we address this gap by developing a generic methodology that economically uses objective function evaluations in a problem-driven manner to prevent optimality gap divergence and avoid explosions in objective evaluations. Our methodology allows for a variety of step size routines and search direction strategies. Furthermore, we develop a particular, novel step size selection methodology that is well-suited to our framework. We show that our specific procedure is highly competitive with standard optimization methods on CUTEst test problems. We then show our specific procedure is highly favorable relative to standard optimization methods on a particularly tough data science problem: learning the parameters in a generalized estimating equation model. Thus, we provide a novel gradient methodology that is better suited to optimization problems from this important class of data science applications.