An overview of domain-specific foundation model: key technologies, applications and challenges

TL;DR

This paper provides a comprehensive overview of how to develop and customize domain-specific foundation models, highlighting key technologies, applications, and future challenges in tailoring general models for specialized industry needs.

Contribution

It offers the first thorough survey of methodologies, architectures, and challenges involved in creating domain-specific foundation models, filling a critical gap in existing literature.

Findings

Survey of key methods for domain-specific FM construction

Discussion of applications across various industries

Identification of main challenges in customization process

Abstract

The impressive performance of ChatGPT and other foundation-model-based products in human language understanding has prompted both academia and industry to explore how these models can be tailored for specific industries and application scenarios. This process, known as the customization of domain-specific foundation models (FMs), addresses the limitations of general-purpose models, which may not fully capture the unique patterns and requirements of domain-specific data. Despite its importance, there is a notable lack of comprehensive overview papers on building domain-specific FMs, while numerous resources exist for general-purpose models. To bridge this gap, this article provides a timely and thorough overview of the methodology for customizing domain-specific FMs. It introduces basic concepts, outlines the general architecture, and surveys key methods for constructing domain-specific models. Furthermore, the article discusses various domains that can benefit from these specialized models and highlights the challenges ahead. Through this overview, we aim to offer valuable guidance and reference for researchers and practitioners from diverse fields to develop their own customized FMs.