Transformer models: an introduction and catalog

TL;DR

This paper provides a comprehensive catalog and classification of popular Transformer models, explaining their key features and innovations, including models trained with self-supervised learning and human-in-the-loop methods.

Contribution

It offers a simple, organized catalog of Transformer models, highlighting their main characteristics and recent advancements in the field.

Findings

Catalog includes models like BERT, GPT-3, and InstructGPT.

Highlights key innovations in Transformer architectures.

Provides a classification framework for Transformer models.

Abstract

In the past few years we have seen the meteoric appearance of dozens of foundation models of the Transformer family, all of which have memorable and sometimes funny, but not self-explanatory, names. The goal of this paper is to offer a somewhat comprehensive but simple catalog and classification of the most popular Transformer models. The paper also includes an introduction to the most important aspects and innovations in Transformer models. Our catalog will include models that are trained using self-supervised learning (e.g., BERT or GPT3) as well as those that are further trained using a human-in-the-loop (e.g. the InstructGPT model used by ChatGPT).