Optimal Deterministic Algorithm Generation

TL;DR

This paper introduces a mathematical programming approach to automatically generate optimal algorithms within a parameterized family, focusing on cost and convergence constraints, demonstrated on nonlinear equations and optimization problems.

Contribution

It presents a novel formulation for deterministic algorithm generation using optimization, incorporating cost and convergence criteria, and demonstrates its effectiveness with prototype implementation.

Findings

Well-known algorithms are shown to be optimal in certain cases.

The formulation can generate algorithms with minimized computational cost.

The approach guarantees optimality using global optimization techniques.

Abstract

A formulation for the automated generation of algorithms via mathematical programming (optimization) is proposed. The formulation is based on the concept of optimizing within a parameterized family of algorithms, or equivalently a family of functions describing the algorithmic steps. The optimization variables are the parameters -within this family of algorithms- that encode algorithm design: the computational steps of which the selected algorithms consists. The objective function of the optimization problem encodes the merit function of the algorithm, e.g., the computational cost (possibly also including a cost component for memory requirements) of the algorithm execution. The constraints of the optimization problem ensure convergence of the algorithm, i.e., solution of the problem at hand. The formulation is described prototypically for algorithms used in solving nonlinear equations and in performing unconstrained optimization; the parametrized algorithm family considered is that of monomials in function and derivative evaluation (including negative powers). A prototype implementation in GAMS is developed along with illustrative results demonstrating cases for which well-known algorithms are shown to be optimal. The formulation is a mixed-integer nonlinear program (MINLP). To overcome the multimodality arising from nonconvexity in the optimization problem, a combination of brute force and general-purpose deterministic global algorithms is employed to guarantee the optimality of the algorithm devised. We then discuss several directions towards which this methodology can be extended, their scope and limitations.