Introduction to Sequence Modeling with Transformers

TL;DR

This paper provides an accessible, step-by-step introduction to transformer architecture, focusing on understanding its core components like tokenization, embedding, and positional encoding through incremental analysis.

Contribution

It offers a simplified, incremental approach to understanding transformer components, emphasizing practical implementation and comprehension of each part's role.

Findings

Clarified the roles of tokenization, embedding, masking, and positional encoding.

Demonstrated how each component affects sequence modeling.

Provided insights into the implementation of transformer components using simple binary sequences.

Abstract

Understanding the transformer architecture and its workings is essential for machine learning (ML) engineers. However, truly understanding the transformer architecture can be demanding, even if you have a solid background in machine learning or deep learning. The main working horse is attention, which yields to the transformer encoder-decoder structure. However, putting attention aside leaves several programming components that are easy to implement but whose role for the whole is unclear. These components are 'tokenization', 'embedding' ('un-embedding'), 'masking', 'positional encoding', and 'padding'. The focus of this work is on understanding them. To keep things simple, the understanding is built incrementally by adding components one by one, and after each step investigating what is doable and what is undoable with the current model. Simple sequences of zeros (0) and ones (1) are used to study the workings of each step.