Low-resource domain adaptation while minimizing energy and hardware resource consumption

TL;DR

This paper investigates energy-efficient methods for domain adaptation of large language models, focusing on low-resource environments by analyzing numerical precision and parallelization strategies to reduce computational costs.

Contribution

It introduces an evaluation of numerical precision formats and data parallelization techniques to enable cost-effective domain adaptation in resource-constrained settings.

Findings

Lower precision formats can reduce training time and energy consumption.

Parallelization strategies impact training speed and model accuracy.

Approaches improve accessibility of domain adaptation for low-resource groups.

Abstract

Training Large Language Models (LLMs) is costly in terms of energy, hardware, and annotated data, often resulting in a positionality rooted in predominant cultures and values (Santy et al., 2023). Domain adaptation has emerged as a promising strategy to better align models with diverse cultural and value contexts (Hershcovich et al., 2022), but its computational cost remains a significant barrier, particularly for research groups lacking access to large-scale infrastructure. In this paper, we evaluate how the use of different numerical precision formats and data parallelization strategies impacts both training speed (as a proxy to energy and hardware consumption) and model accuracy, with the goal of facilitating domain adaptation in low-resource environments. Our findings are relevant to any setting where energy efficiency, accessibility, or limited hardware availability are key concerns.