A Survey on Memory-Efficient Transformer-Based Model Training in AI for Science

TL;DR

This paper surveys memory-efficient transformer training techniques for large language models in scientific research, highlighting methods to reduce memory use while maintaining accuracy, thus enabling scalable AI applications in science.

Contribution

It systematically categorizes and reviews various memory-efficient pre-training techniques for large-scale transformers used in scientific domains.

Findings

Memory optimization methods can significantly reduce storage requirements.

Tailored techniques maintain prediction accuracy while improving efficiency.

Bridging efficiency and scientific needs enables scalable AI in science.

Abstract

Scientific research faces high costs and inefficiencies with traditional methods, but the rise of deep learning and large language models (LLMs) offers innovative solutions. This survey reviews transformer-based LLM applications across scientific fields such as biology, medicine, chemistry, and meteorology, underscoring their role in advancing research. However, the continuous expansion of model size has led to significant memory demands, hindering further development and application of LLMs for science. This survey systematically reviews and categorizes memory-efficient pre-training techniques for large-scale transformers, including algorithm-level, system-level, and hardware-software co-optimization. Using AlphaFold 2 as an example, we demonstrate how tailored memory optimization methods can reduce storage needs while preserving prediction accuracy. By bridging model efficiency and scientific application needs, we hope to provide insights for scalable and cost-effective LLM training in AI for science.