Surrogate Ensemble in Expensive Multi-Objective Optimization via Deep Q-Learning

TL;DR

This paper introduces SEEMOO, a reinforcement learning-based ensemble framework that dynamically selects surrogate models during expensive multi-objective optimization, significantly improving performance over single-surrogate methods.

Contribution

The paper presents a novel RL-assisted ensemble approach for surrogate model selection in multi-objective optimization, reducing human bias and enhancing robustness and efficiency.

Findings

SEEMOO outperforms single-surrogate baselines in benchmark tests.

The ensemble approach improves optimization robustness across diverse problems.

Ablation studies confirm the effectiveness of SEEMOO's components.

Abstract

Surrogate-assisted Evolutionary Algorithms~(SAEAs) have shown promising robustness in solving expensive optimization problems. A key aspect that impacts SAEAs' effectiveness is surrogate model selection, which in existing works is predominantly decided by human developer. Such human-made design choice introduces strong bias into SAEAs and may hurt their expected performance on out-of-scope tasks. In this paper, we propose a reinforcement learning-assisted ensemble framework, termed as SEEMOO, which is capable of scheduling different surrogate models within a single optimization process, hence boosting the overall optimization performance in a cooperative paradigm. Specifically, we focus on expensive multi-objective optimization problems, where multiple objective functions shape a compositional landscape and hence challenge surrogate selection. SEEMOO comprises following core designs: 1) A pre-collected model pool that maintains different surrogate models; 2) An attention-based state-extractor supports universal optimization state representation of problems with varied objective numbers; 3) a deep Q-network serves as dynamic surrogate selector: Given the optimization state, it selects desired surrogate model for current-step evaluation. SEEMOO is trained to maximize the overall optimization performance under a training problem distribution. Extensive benchmark results demonstrate SEEMOO's surrogate ensemble paradigm boosts the optimization performance of single-surrogate baselines. Further ablation studies underscore the importance of SEEMOO's design components.