Practical recommendations for gradient-based training of deep architectures

TL;DR

This paper provides practical guidance on hyper-parameter tuning and training strategies for deep neural networks using gradient-based methods, emphasizing efficiency and debugging in large-scale models.

Contribution

It offers concrete recommendations for hyper-parameter choices and discusses practical issues in training deep architectures, including open questions about training difficulties.

Findings

Effective hyper-parameter adjustment improves training outcomes.

Guidelines for debugging large-scale deep networks.

Identification of open challenges in training very deep architectures.

Abstract

Learning algorithms related to artificial neural networks and in particular for Deep Learning may seem to involve many bells and whistles, called hyper-parameters. This chapter is meant as a practical guide with recommendations for some of the most commonly used hyper-parameters, in particular in the context of learning algorithms based on back-propagated gradient and gradient-based optimization. It also discusses how to deal with the fact that more interesting results can be obtained when allowing one to adjust many hyper-parameters. Overall, it describes elements of the practice used to successfully and efficiently train and debug large-scale and often deep multi-layer neural networks. It closes with open questions about the training difficulties observed with deeper architectures.