Thunder-LLM: Efficiently Adapting LLMs to Korean with Minimal Resources

TL;DR

This paper introduces Thunder-LLM, a cost-effective method for adapting English-based large language models to Korean, demonstrating superior performance with minimal data and resources.

Contribution

It presents a complete end-to-end process for low-resource language adaptation of LLMs, including data collection, training, and evaluation, with publicly available code.

Findings

Thunder-LLM models outperform existing Korean LLMs

Effective adaptation achieved with minimal data and compute

Comprehensive methodology shared for low-resource language adaptation

Abstract

Since state-of-the-art LLMs often underperform in languages other than English or Chinese, improving the capability of LLMs in new languages has become an essential task. Moreover, LLMs' entire end-to-end training process remains largely unknown to the public due to proprietary reasons, technical complexity, inconsistent documentation, and ethical considerations. The complete picture remains a closely guarded secret within the industry. This paper presents methods to adapt an existing English-based LLM to Korean in a low-budget scenario. We describe the entire end-to-end process: collecting Korean datasets, preprocessing the data, training the model, creating downstream benchmarks, and conducting evaluations. The evaluation results indicate that our method can effectively and cost-efficiently add new language capabilities to existing LLMs. Our new bilingual models, Thunder-LLM and Thunder-LLM-Ins, achieve superior Korean performance compared to state-of-the-art models while utilizing minimal data and computational resources. We share our comprehensive experience and make the code publicly available.