Optimization via Strategic Law of Large Numbers

TL;DR

This paper introduces a unified theoretical framework for global optimization based on the Strategic Law of Large Numbers, and proposes the SMCO algorithms that efficiently find global maxima in complex landscapes with minimal starting points.

Contribution

The paper develops a novel theoretical approach linking decision strategies to optimization convergence, and introduces the SMCO algorithms that outperform existing methods in challenging scenarios.

Findings

SMCO algorithms reliably find global maxima in high-dimensional problems.

Theoretical convergence guarantees for strategies based on the Strategic Law of Large Numbers.

Empirical results show SMCO outperforms state-of-the-art optimizers on complex functions.

Abstract

This paper proposes a unified framework for the global optimization of a continuous function in a bounded rectangular domain. Specifically, we show that: (1) under the optimal strategy for a two-armed decision model, the sample mean converges to a global optimizer under the Strategic Law of Large Numbers, and (2) a sign-based strategy built upon the solution of a parabolic PDE is asymptotically optimal. Motivated by this result, we propose a class of {\bf S}trategic {\bf M}onte {\bf C}arlo {\bf O}ptimization (SMCO) algorithms, which uses a simple strategy that makes coordinate-wise two-armed decisions based on the signs of the partial gradient of the original function being optimized over (without the need of solving PDEs). While this simple strategy is not generally optimal, we show that it is sufficient for our SMCO algorithm to converge to local optimizer(s) from a single starting point, and to global optimizers under a growing set of starting points. Numerical studies demonstrate the suitability of our SMCO algorithms for global optimization, and illustrate the promise of our theoretical framework and practical approach. For a wide range of test functions with challenging optimization landscapes (including ReLU neural networks with square and hinge loss), our SMCO algorithms converge to the global maximum accurately and robustly, using only a small set of starting points (at most 100 for dimensions up to 1000) and a small maximum number of iterations (200). In fact, our algorithms outperform many state-of-the-art global optimizers, as well as local algorithms augmented with the same set of starting points as ours.