Kanana: Compute-efficient Bilingual Language Models

TL;DR

Kanana introduces a series of compute-efficient bilingual language models optimized for Korean and English, achieving high performance with lower computational costs through innovative training and fine-tuning techniques.

Contribution

The paper presents a new series of bilingual models with reduced computational costs and detailed methods for training, fine-tuning, and adapting models for specific tasks.

Findings

Exceeds performance in Korean

Competitive in English

Significantly lower computational cost

Abstract

We introduce Kanana, a series of bilingual language models that demonstrate exceeding performance in Korean and competitive performance in English. The computational cost of Kanana is significantly lower than that of state-of-the-art models of similar size. The report details the techniques employed during pre-training to achieve compute-efficient yet competitive models, including high quality data filtering, staged pre-training, depth up-scaling, and pruning and distillation. Furthermore, the report outlines the methodologies utilized during the post-training of the Kanana models, encompassing supervised fine-tuning and preference optimization, aimed at enhancing their capability for seamless interaction with users. Lastly, the report elaborates on plausible approaches used for language model adaptation to specific scenarios, such as embedding, retrieval augmented generation, and function calling. The Kanana model series spans from 2.1B to 32.5B parameters with 2.1B models (base, instruct, embedding) publicly released to promote research on Korean language models.