A Surrogate-Assisted Highly Cooperative Coevolutionary Algorithm for Hyperparameter Optimization in Deep Convolutional Neural Network

TL;DR

This paper introduces SHCHO, a surrogate-assisted cooperative hyperparameter optimization method that decomposes CNNs into subunits for efficient hyperparameter tuning, significantly improving CNN performance with reduced computational cost.

Contribution

The paper presents a novel surrogate-assisted cooperative hyperparameter optimization algorithm that decomposes CNNs into subunits and employs cooperation mechanisms to enhance hyperparameter tuning.

Findings

SHCHO effectively improves CNN performance on image classification tasks.

The surrogate-assisted approach reduces computational costs significantly.

Cooperative optimization outperforms traditional hyperparameter tuning methods.

Abstract

Convolutional neural networks (CNNs) have gained remarkable success in recent years. However, their performance highly relies on the architecture hyperparameters, and finding proper hyperparameters for a deep CNN is a challenging optimization problem owing to its high-dimensional and computationally expensive characteristics. Given these difficulties, this study proposes a surrogate-assisted highly cooperative hyperparameter optimization (SHCHO) algorithm for chain-styled CNNs. To narrow the large search space, SHCHO first decomposes the whole CNN into several overlapping sub-CNNs in accordance with the overlapping hyperparameter interaction structure and then cooperatively optimizes these hyperparameter subsets. Two cooperation mechanisms are designed during this process. One coordinates all the sub-CNNs to reproduce the information flow in the whole CNN and achieve macro cooperation among them, and the other tackles the overlapping components by simultaneously considering the involved two sub-CNNs and facilitates micro cooperation between them. As a result, a proper hyperparameter configuration can be effectively located for the whole CNN. Besides, SHCHO also employs the well-performing surrogate technique to assist in the hyperparameter optimization of each sub-CNN, thereby greatly reducing the expensive computational cost. Extensive experimental results on two widely-used image classification datasets indicate that SHCHO can significantly improve the performance of CNNs.