Better as Generators Than Classifiers: Leveraging LLMs and Synthetic Data for Low-Resource Multilingual Classification

TL;DR

This paper demonstrates that large language models are more effective as data generators than classifiers, enabling smaller models to outperform the large models themselves in low-resource multilingual classification tasks.

Contribution

The study introduces a synthetic data generation approach using LLMs to train smaller models, showing improved performance in low-resource multilingual classification.

Findings

Small synthetic datasets improve smaller model performance.

Synthetic data enables smaller models to outperform large LLMs.

LLMs are more effective as data generators than classifiers.

Abstract

Large Language Models (LLMs) have demonstrated remarkable multilingual capabilities, making them promising tools in both high- and low-resource languages. One particularly valuable use case is generating synthetic samples that can be used to train smaller models in low-resource scenarios where human-labelled data is scarce. In this work, we investigate whether these synthetic data generation capabilities can serve as a form of distillation, producing smaller models that perform on par with or even better than massive LLMs across languages and tasks. To this end, we use a state-of-the-art multilingual LLM to generate synthetic datasets covering 11 languages and 4 classification tasks. These datasets are then used to train smaller models via fine-tuning or instruction tuning, or as synthetic in-context examples for compact LLMs. Our experiments show that even small amounts of synthetic data enable smaller models to outperform the large generator itself, particularly in low-resource languages. Overall, the results suggest that LLMs are best utilised as generators (teachers) rather than classifiers, producing data that empowers smaller and more efficient multilingual models.