Pre-training Attention Mechanisms

TL;DR

This paper introduces a new method called bootstrapped glimpse mimicking that improves training of attention mechanisms in neural networks, leading to better performance and efficiency, inspired by human learning processes.

Contribution

It presents a novel, task-general training approach for attention models that enhances performance and efficiency without requiring explicit supervision.

Findings

Pretraining on unsupervised tasks helps discover effective attention policies.

Guided attention improves classification performance.

Increased attention efficiency contributes to better results.

Abstract

Recurrent neural networks with differentiable attention mechanisms have had success in generative and classification tasks. We show that the classification performance of such models can be enhanced by guiding a randomly initialized model to attend to salient regions of the input in early training iterations. We further show that, if explicit heuristics for guidance are unavailable, a model that is pretrained on an unsupervised reconstruction task can discover good attention policies without supervision. We demonstrate that increased efficiency of the attention mechanism itself contributes to these performance improvements. Based on these insights, we introduce bootstrapped glimpse mimicking, a simple, theoretically task-general method of more effectively training attention models. Our work draws inspiration from and parallels results on human learning of attention.