Designing Large Foundation Models for Efficient Training and Inference: A Survey

TL;DR

This survey reviews modern techniques for efficient training and inference of large foundation models, emphasizing model and system design to reduce computational costs and improve accessibility.

Contribution

It provides a comprehensive overview of current methods in model and system design for efficient foundation models, including a curated repository.

Findings

Highlights key techniques for efficient LLM training and inference

Identifies challenges and future directions in model and system optimization

Provides a resource repository for further research

Abstract

This paper focuses on modern efficient training and inference technologies on foundation models and illustrates them from two perspectives: model and system design. Model and System Design optimize LLM training and inference from different aspects to save computational resources, making LLMs more efficient, affordable, and more accessible. The paper list repository is available at https://github.com/NoakLiu/Efficient-Foundation-Models-Survey.