Learning Wake-Sleep Recurrent Attention Models

TL;DR

This paper introduces the Wake-Sleep Recurrent Attention Model, a training method for stochastic attention networks that enhances inference, reduces gradient variance, and accelerates training for image classification and captioning tasks.

Contribution

It presents a novel training approach for stochastic attention models that improves posterior inference and reduces gradient variance, leading to faster training.

Findings

Significantly speeds up training of stochastic attention networks.

Improves posterior inference in attention-based models.

Effective in image classification and caption generation.

Abstract

Despite their success, convolutional neural networks are computationally expensive because they must examine all image locations. Stochastic attention-based models have been shown to improve computational efficiency at test time, but they remain difficult to train because of intractable posterior inference and high variance in the stochastic gradient estimates. Borrowing techniques from the literature on training deep generative models, we present the Wake-Sleep Recurrent Attention Model, a method for training stochastic attention networks which improves posterior inference and which reduces the variability in the stochastic gradients. We show that our method can greatly speed up the training time for stochastic attention networks in the domains of image classification and caption generation.