RedWhale: An Adapted Korean LLM Through Efficient Continual Pretraining

TL;DR

RedWhale is a Korean-specific large language model developed through efficient continual pretraining, leveraging cross-lingual transfer to outperform existing models on Korean NLP benchmarks while reducing training costs.

Contribution

The paper introduces RedWhale, a novel Korean LLM built with an efficient pretraining strategy and cross-lingual transfer, addressing resource constraints and improving Korean NLP performance.

Findings

RedWhale outperforms existing models on Korean benchmarks.

Pretraining on 9.7 billion tokens shows no signs of convergence.

Efficient strategies reduce training time and computational costs.

Abstract

The field of Natural Language Processing (NLP) has seen significant advancements with the development of Large Language Models (LLMs). However, much of this research remains focused on English, often overlooking low-resource languages like Korean. This oversight presents challenges due to the unique non-alphabetic token structure of Korean and the substantial memory and computational demands required for LLM training, which frequently lead to memory constraints and out-of-memory errors. To address these issues, we present RedWhale, a model specifically tailored for Korean language processing. RedWhale is developed using an efficient continual pretraining approach that includes a comprehensive Korean corpus preprocessing pipeline, a specialized tokenizer, an optimized model initialization technique, and a multistage pretraining strategy. These innovations collectively reduce training time and computational costs while maintaining high levels of accuracy and comprehension. By leveraging cross-lingual transfer learning, which exploits shared linguistic similarities across languages, RedWhale builds on English models to enhance Korean language processing. Experimental results demonstrate that RedWhale outperforms other leading models on Korean NLP benchmarks, including the Korean Balanced Evaluation of Significant Tasks (KoBEST), showing superior understanding and generation of Korean text. Furthermore, RedWhale showed no signs of convergence even after pretraining on 9.7 billion tokens, indicating the potential for further improvements with additional training. This work represents a significant advancement in bridging the linguistic divide, particularly in enhancing NLP capabilities for the Korean language.