Hyperparameters Optimization in Deep Convolutional Neural Network / Bayesian Approach with Gaussian Process Prior

TL;DR

This paper discusses the use of Bayesian Optimization with Gaussian Process priors for hyperparameter tuning in deep convolutional neural networks, aiming to improve efficiency over traditional search methods.

Contribution

It provides an overview of the mathematical foundation of Bayesian Optimization with Gaussian processes applied to hyperparameter tuning in ConvNets.

Findings

Bayesian Optimization outperforms grid and random search methods.

Recent Bayesian approaches achieved lower error rates on CIFAR-10.

Gaussian process priors effectively model hyperparameter search space.

Abstract

Convolutional Neural Network is known as ConvNet have been extensively used in many complex machine learning tasks. However, hyperparameters optimization is one of a crucial step in developing ConvNet architectures, since the accuracy and performance are reliant on the hyperparameters. This multilayered architecture parameterized by a set of hyperparameters such as the number of convolutional layers, number of fully connected dense layers & neurons, the probability of dropout implementation, learning rate. Hence the searching the hyperparameter over the hyperparameter space are highly difficult to build such complex hierarchical architecture. Many methods have been proposed over the decade to explore the hyperparameter space and find the optimum set of hyperparameter values. Reportedly, Gird search and Random search are said to be inefficient and extremely expensive, due to a large number of hyperparameters of the architecture. Hence, Sequential model-based Bayesian Optimization is a promising alternative technique to address the extreme of the unknown cost function. The recent study on Bayesian Optimization by Snoek in nine convolutional network parameters is achieved the lowerest error report in the CIFAR-10 benchmark. This article is intended to provide the overview of the mathematical concept behind the Bayesian Optimization over a Gaussian prior.