An Equivalent Circuit Workflow for Unconstrained Optimization

TL;DR

This paper presents a novel workflow for unconstrained optimization by mapping objective functions onto an equivalent circuit model, enabling the design of robust, fast-converging algorithms leveraging circuit domain knowledge.

Contribution

It introduces a circuit-based framework for unconstrained optimization, allowing the derivation of existing and new algorithms with improved robustness and convergence properties.

Findings

Existing descent algorithms are special cases of the circuit-based approach.

New algorithms derived from the model show robustness to hyperparameters.

The approach achieves convergence rates comparable or superior to state-of-the-art methods.

Abstract

We introduce a new workflow for unconstrained optimization whereby objective functions are mapped onto a physical domain to more easily design algorithms that are robust to hyperparameters and achieve fast convergence rates. Specifically, we represent optimization problems as an equivalent circuit that are then solved solely as nonlinear circuits using robust solution methods. The equivalent circuit models the trajectory of component-wise scaled gradient flow problem as the transient response of the circuit for which the steady-state coincides with a critical point of the objective function. The equivalent circuit model leverages circuit domain knowledge to methodically design new optimization algorithms that would likely not be developed without a physical model. We incorporate circuit knowledge into optimization methods by 1) enhancing the underlying circuit model for fast numerical analysis, 2) controlling the optimization trajectory by designing the nonlinear circuit components, and 3) solving for step sizes using well-known methods from the circuit simulation. We first establish the necessary conditions that the controls must fulfill for convergence. We show that existing descent algorithms can be re-derived as special cases of this approach and derive new optimization algorithms that are developed with insights from a circuit-based model. The new algorithms can be designed to be robust to hyperparameters, achieve convergence rates comparable or faster than state of the art methods, and are applicable to optimizing a variety of both convex and nonconvex problems.