On the Learning Dynamics of Attention Networks

TL;DR

This paper investigates the learning dynamics of attention networks under different loss functions, revealing distinct behaviors and proposing a hybrid approach that leverages their advantages for improved performance.

Contribution

It provides a theoretical analysis of attention learning dynamics, derives closed-form parameter trajectories, and introduces a hybrid loss method that enhances attention model training.

Findings

Soft attention improves quickly initially but stagnates later.

Hard attention shows the opposite pattern, improving slowly at first.

A hybrid approach combining both losses outperforms individual paradigms.

Abstract

Attention models are typically learned by optimizing one of three standard loss functions that are variously called -- soft attention, hard attention, and latent variable marginal likelihood (LVML) attention. All three paradigms are motivated by the same goal of finding two models -- a `focus' model that `selects' the right \textit{segment} of the input and a `classification' model that processes the selected segment into the target label. However, they differ significantly in the way the selected segments are aggregated, resulting in distinct dynamics and final results. We observe a unique signature of models learned using these paradigms and explain this as a consequence of the evolution of the classification model under gradient descent when the focus model is fixed. We also analyze these paradigms in a simple setting and derive closed-form expressions for the parameter trajectory under gradient flow. With the soft attention loss, the focus model improves quickly at initialization and splutters later on. On the other hand, hard attention loss behaves in the opposite fashion. Based on our observations, we propose a simple hybrid approach that combines the advantages of the different loss functions and demonstrates it on a collection of semi-synthetic and real-world datasets