Training a T5 Using Lab-sized Resources

TL;DR

This paper introduces techniques and practical recommendations for training large language models like T5 with limited resources, demonstrated through a Danish T5 case study, making advanced NLP more accessible.

Contribution

It presents novel methods and guidelines enabling training of large models on modest resources, including a first Danish T5 model case study.

Findings

Achieved training of a large T5 model with limited resources

Provided practical recommendations for resource-constrained training

Developed the first Danish T5 model

Abstract

Training large neural language models on large datasets is resource- and time-intensive. These requirements create a barrier to entry, where those with fewer resources cannot build competitive models. This paper presents various techniques for making it possible to (a) train a large language model using resources that a modest research lab might have, and (b) train it in a reasonable amount of time. We provide concrete recommendations for practitioners, which we illustrate with a case study: a T5 model for Danish, the first for this language.