Language Models: A Guide for the Perplexed

TL;DR

This paper provides an accessible overview of language models, clarifying their scientific basis, development history, and current understanding to improve AI literacy among the public and educators.

Contribution

It offers a non-technical, research-focused explanation of language models, distinguishing them from products and human cognition, and situates them within their scientific context.

Findings

Clarifies the scientific perspective on language models

Distinguishes models from AI products and human cognition

Summarizes current knowledge boundaries

Abstract

Given the growing importance of AI literacy, we decided to write this tutorial to help narrow the gap between the discourse among those who study language models -- the core technology underlying ChatGPT and similar products -- and those who are intrigued and want to learn more about them. In short, we believe the perspective of researchers and educators can add some clarity to the public's understanding of the technologies beyond what's currently available, which tends to be either extremely technical or promotional material generated about products by their purveyors. Our approach teases apart the concept of a language model from products built on them, from the behaviors attributed to or desired from those products, and from claims about similarity to human cognition. As a starting point, we (1) offer a scientific viewpoint that focuses on questions amenable to study through experimentation; (2) situate language models as they are today in the context of the research that led to their development; and (3) describe the boundaries of what is known about the models at this writing.