Taiwan LLM: Bridging the Linguistic Divide with a Culturally Aligned Language Model

TL;DR

Taiwan LLM is a culturally aligned language model specifically designed for Traditional Chinese as used in Taiwan, achieving superior understanding and generation by incorporating linguistic and cultural nuances.

Contribution

It introduces the first culturally resonant Large Language Model for Traditional Chinese, tailored to Taiwanese linguistic and cultural specifics, with open-source resources.

Findings

Outperforms existing models on Traditional Chinese tasks

Achieves high accuracy in understanding and generating Taiwanese Traditional Chinese

Open-source release fosters further research and collaboration

Abstract

In the realm of language models, the nuanced linguistic and cultural intricacies of Traditional Chinese, as spoken in Taiwan, have been largely overlooked. This paper introduces Taiwan LLM, a pioneering Large Language Model that specifically caters to the Traditional Chinese language, with a focus on the variant used in Taiwan. Leveraging a comprehensive pretraining corpus and instruction-finetuning datasets, we have developed a model that not only understands the complexities of Traditional Chinese but also embodies the cultural context of Taiwan. Taiwan LLM represents the first of its kind, a model that is not only linguistically accurate but also culturally resonant with its user base. Our evaluations demonstrate that Taiwan LLM achieves superior performance in understanding and generating Traditional Chinese text, outperforming existing models that are predominantly trained on Simplified Chinese or English. The open-source release of Taiwan LLM invites collaboration and further innovation, ensuring that the linguistic diversity of Chinese speakers is embraced and well-served. The model, datasets, and further resources are made publicly available to foster ongoing research and development in this field.