Towards Constituting Mathematical Structures for Learning to Optimize

TL;DR

This paper introduces a mathematically grounded structure for Learning to Optimize models, aiming to improve generalization and robustness across diverse optimization problems, validated through theoretical analysis and numerical experiments.

Contribution

It derives fundamental mathematical conditions for effective update rules and proposes a new L2O model with a structure inspired by these conditions, enhancing generalization.

Findings

The proposed model outperforms existing L2O methods in out-of-distribution tests.

Mathematical conditions are validated through numerical simulations.

The structure improves robustness and applicability of L2O algorithms.

Abstract

Learning to Optimize (L2O), a technique that utilizes machine learning to learn an optimization algorithm automatically from data, has gained arising attention in recent years. A generic L2O approach parameterizes the iterative update rule and learns the update direction as a black-box network. While the generic approach is widely applicable, the learned model can overfit and may not generalize well to out-of-distribution test sets. In this paper, we derive the basic mathematical conditions that successful update rules commonly satisfy. Consequently, we propose a novel L2O model with a mathematics-inspired structure that is broadly applicable and generalized well to out-of-distribution problems. Numerical simulations validate our theoretical findings and demonstrate the superior empirical performance of the proposed L2O model.