Foundation models in brief: A historical, socio-technical focus
Johannes Schneider

TL;DR
This paper provides an overview of foundation models, highlighting their historical development, socio-technical implications, emergent behaviors like in-context learning, and potential shifts in AI power dynamics.
Contribution
It offers a clear distinction between foundation models and previous models, discusses socio-technical aspects, and explores future research directions.
Findings
Foundation models achieve state-of-the-art performance across domains.
Emergent behaviors like in-context learning enable few-shot adaptation.
Homogenization may centralize AI control among few corporations.
Abstract
Foundation models can be disruptive for future AI development by scaling up deep learning in terms of model size and training data's breadth and size. These models achieve state-of-the-art performance (often through further adaptation) on a variety of tasks in domains such as natural language processing and computer vision. Foundational models exhibit a novel {emergent behavior}: {In-context learning} enables users to provide a query and a few examples from which a model derives an answer without being trained on such queries. Additionally, {homogenization} of models might replace a myriad of task-specific models with fewer very large models controlled by few corporations leading to a shift in power and control over AI. This paper provides a short introduction to foundation models. It contributes by crafting a crisp distinction between foundation models and prior deep learning models,…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMachine Learning and Data Classification
