Multimodal Foundation Models: From Specialists to General-Purpose Assistants

TL;DR

This survey reviews the evolution of multimodal foundation models from specialized systems to versatile, general-purpose assistants, highlighting recent advances in unified models, multimodal LLMs, and tool chaining.

Contribution

It provides a comprehensive taxonomy and analysis of both established and emerging research areas in multimodal foundation models, emphasizing their transition to general-purpose AI assistants.

Findings

Survey of well-established multimodal models for visual understanding and text-to-image generation.

Analysis of recent advances in unified vision models inspired by large language models.

Discussion of end-to-end training of multimodal LLMs and chaining multimodal tools.

Abstract

This paper presents a comprehensive survey of the taxonomy and evolution of multimodal foundation models that demonstrate vision and vision-language capabilities, focusing on the transition from specialist models to general-purpose assistants. The research landscape encompasses five core topics, categorized into two classes. (i) We start with a survey of well-established research areas: multimodal foundation models pre-trained for specific purposes, including two topics -- methods of learning vision backbones for visual understanding and text-to-image generation. (ii) Then, we present recent advances in exploratory, open research areas: multimodal foundation models that aim to play the role of general-purpose assistants, including three topics -- unified vision models inspired by large language models (LLMs), end-to-end training of multimodal LLMs, and chaining multimodal tools with LLMs. The target audiences of the paper are researchers, graduate students, and professionals in computer vision and vision-language multimodal communities who are eager to learn the basics and recent advances in multimodal foundation models.