Kakugo: Distillation of Low-Resource Languages into Small Language Models

TL;DR

Kakugo introduces a cost-effective pipeline for training small language models for low-resource languages using synthetic data generated by large teacher models, enabling accessible language-specific AI development.

Contribution

The paper presents a novel pipeline that efficiently trains small language models for 54 low-resource languages using only language names and synthetic data.

Findings

Performance improvements across NLP tasks

Cost-effective training under $50 per language

Effective for diverse low-resource languages

Abstract

We present Kakugo, a novel and cost-effective pipeline designed to train general-purpose Small Language Models (SLMs) for low-resource languages using only the language name as input. By using a large teacher model to generate synthetic prompts and translate instruction datasets, we produced training data and SLMs for 54 low-resource languages. Evaluations across a diverse set of general natural language processing tasks, including translation, classification, and question answering, demonstrate that our pipeline consistently improves performance over base models. With a total generation and training cost of under $50 per language, Kakugo offers an accessible method for communities to develop language-specific AI.