Convergence rates for pretraining and dropout: Guiding learning parameters using network structure

TL;DR

This paper investigates how network structure influences the convergence rates of pretraining and dropout in deep learning, providing theoretical bounds and practical insights for parameter choices.

Contribution

It introduces a framework linking network structure and learning parameters to convergence rates, incorporating denoising autoencoder and dropout mechanisms.

Findings

Derived convergence rate bounds for deep networks with dropout and autoencoders

Provided guidelines for selecting learning parameters based on network structure

Supported theoretical results with empirical evaluations

Abstract

Unsupervised pretraining and dropout have been well studied, especially with respect to regularization and output consistency. However, our understanding about the explicit convergence rates of the parameter estimates, and their dependence on the learning (like denoising and dropout rate) and structural (like depth and layer lengths) aspects of the network is less mature. An interesting question in this context is to ask if the network structure could "guide" the choices of such learning parameters. In this work, we explore these gaps between network structure, the learning mechanisms and their interaction with parameter convergence rates. We present a way to address these issues based on the backpropagation convergence rates for general nonconvex objectives using first-order information. We then incorporate two learning mechanisms into this general framework -- denoising autoencoder and dropout, and subsequently derive the convergence rates of deep networks. Building upon these bounds, we provide insights into the choices of learning parameters and network sizes that achieve certain levels of convergence accuracy. The results derived here support existing empirical observations, and we also conduct a set of experiments to evaluate them.