Learning Mixture-of-Experts for General-Purpose Black-Box Discrete Optimization

TL;DR

MEGO is a data-driven, mixture-of-experts neural optimizer that generalizes across various discrete optimization problems, outperforming existing methods in quality and efficiency, and offers insights into problem similarity.

Contribution

Introduces MEGO, a novel mixture-of-experts neural optimizer trained via learning-to-optimize, capable of solving diverse discrete problems with high efficiency and quality.

Findings

MEGO outperforms traditional general-purpose optimizers in multiple problem classes.

MEGO surpasses some specialized state-of-the-art optimizers.

Provides a new problem similarity measure for classification.

Abstract

Real-world applications involve various discrete optimization problems. Designing a specialized optimizer for each of these problems is challenging, typically requiring significant domain knowledge and human efforts. Hence, developing general-purpose optimizers as an off-the-shelf tool for a wide range of problems has been a long-standing research target. This article introduces MEGO, a novel general-purpose neural optimizer trained through a fully data-driven learning-to-optimize (L2O) approach. MEGO consists of a mixture-of-experts trained on experiences from solving training problems and can be viewed as a foundation model for optimization problems with binary decision variables. When presented with a problem to solve, MEGO actively selects relevant expert models to generate high-quality solutions. MEGO can be used as a standalone sample-efficient optimizer or in conjunction with existing search methods as an initial solution generator. The generality of MEGO is validated across six problem classes, including three classic problem classes and three problem classes arising from real-world applications in compilers, network analysis, and 3D reconstruction. Trained solely on classic problem classes, MEGO performs very well on all six problem classes, significantly surpassing widely used general-purpose optimizers in both solution quality and efficiency. In some cases, MEGO even surpasses specialized state-of-the-art optimizers. Additionally, MEGO provides a similarity measure between problems, yielding a new perspective for problem classification. In the pursuit of general-purpose optimizers through L2O, MEGO represents an initial yet significant step forward.