Efficient Hyperparameter Optimization in Deep Learning Using a Variable Length Genetic Algorithm

TL;DR

This paper introduces a variable length genetic algorithm to efficiently optimize CNN hyperparameters, addressing the challenge of variable model depth and hyperparameter count in deep learning.

Contribution

The paper presents a novel variable length genetic algorithm tailored for hyperparameter tuning in CNNs, improving search efficiency over fixed-length methods.

Findings

The algorithm finds good hyperparameters efficiently.

More optimization time leads to better CNN performance.

The approach adapts to variable hyperparameter counts based on model depth.

Abstract

Convolutional Neural Networks (CNN) have gained great success in many artificial intelligence tasks. However, finding a good set of hyperparameters for a CNN remains a challenging task. It usually takes an expert with deep knowledge, and trials and errors. Genetic algorithms have been used in hyperparameter optimizations. However, traditional genetic algorithms with fixed-length chromosomes may not be a good fit for optimizing deep learning hyperparameters, because deep learning models have variable number of hyperparameters depending on the model depth. As the depth increases, the number of hyperparameters grows exponentially, and searching becomes exponentially harder. It is important to have an efficient algorithm that can find a good model in reasonable time. In this article, we propose to use a variable length genetic algorithm (GA) to systematically and automatically tune the hyperparameters of a CNN to improve its performance. Experimental results show that our algorithm can find good CNN hyperparameters efficiently. It is clear from our experiments that if more time is spent on optimizing the hyperparameters, better results could be achieved. Theoretically, if we had unlimited time and CPU power, we could find the optimized hyperparameters and achieve the best results in the future.