Transformers from an Optimization Perspective

TL;DR

This paper investigates whether Transformers can be viewed as an optimization process by identifying an underlying energy function, providing a new interpretability perspective for these complex models.

Contribution

It introduces techniques to associate energy function minimization with Transformer layers, bridging the gap in understanding self-attention models from an optimization perspective.

Findings

Established a connection between energy minimization and Transformer layers

Demonstrated the feasibility of interpreting Transformers as unfolding optimization processes

Provided foundational techniques to analyze complex deep models through an energy function lens

Abstract

Deep learning models such as the Transformer are often constructed by heuristics and experience. To provide a complementary foundation, in this work we study the following problem: Is it possible to find an energy function underlying the Transformer model, such that descent steps along this energy correspond with the Transformer forward pass? By finding such a function, we can view Transformers as the unfolding of an interpretable optimization process across iterations. This unfolding perspective has been frequently adopted in the past to elucidate more straightforward deep models such as MLPs and CNNs; however, it has thus far remained elusive obtaining a similar equivalence for more complex models with self-attention mechanisms like the Transformer. To this end, we first outline several major obstacles before providing companion techniques to at least partially address them, demonstrating for the first time a close association between energy function minimization and deep layers with self-attention. This interpretation contributes to our intuition and understanding of Transformers, while potentially laying the ground-work for new model designs.