The Cost of Training NLP Models: A Concise Overview

TL;DR

This paper reviews the factors influencing the high costs of training large NLP models, providing insights for practitioners and non-experts to understand the economic aspects of modern NLP development.

Contribution

It offers a concise overview of the main cost drivers in training large-scale NLP models, highlighting key economic considerations.

Findings

Identification of major cost drivers in NLP training

Analysis of economic factors affecting model development

Guidance for budgeting and resource allocation in NLP projects

Abstract

We review the cost of training large-scale language models, and the drivers of these costs. The intended audience includes engineers and scientists budgeting their model-training experiments, as well as non-practitioners trying to make sense of the economics of modern-day Natural Language Processing (NLP).