Foundation models in brief: A historical, socio-technical focus

TL;DR

This paper provides an overview of foundation models, highlighting their historical development, socio-technical implications, emergent behaviors like in-context learning, and potential shifts in AI power dynamics.

Contribution

It offers a clear distinction between foundation models and previous models, discusses socio-technical aspects, and explores future research directions.

Findings

Foundation models achieve state-of-the-art performance across domains.

Emergent behaviors like in-context learning enable few-shot adaptation.

Homogenization may centralize AI control among few corporations.

Abstract

Foundation models can be disruptive for future AI development by scaling up deep learning in terms of model size and training data's breadth and size. These models achieve state-of-the-art performance (often through further adaptation) on a variety of tasks in domains such as natural language processing and computer vision. Foundational models exhibit a novel {emergent behavior}: {In-context learning} enables users to provide a query and a few examples from which a model derives an answer without being trained on such queries. Additionally, {homogenization} of models might replace a myriad of task-specific models with fewer very large models controlled by few corporations leading to a shift in power and control over AI. This paper provides a short introduction to foundation models. It contributes by crafting a crisp distinction between foundation models and prior deep learning models, providing a history of machine learning leading to foundation models, elaborating more on socio-technical aspects, i.e., organizational issues and end-user interaction, and a discussion of future research.