TF3-RO-50M: Training Compact Romanian Language Models from Scratch on Synthetic Moral Microfiction

TL;DR

This paper presents TF3-RO, a comprehensive pipeline for training compact Romanian language models from scratch using synthetic data, including tokenizer design, model training, compression, and evaluation, tailored for morphologically rich Romanian.

Contribution

It introduces a reproducible, linguistically informed framework for developing Romanian language models and generating synthetic narratives, filling a gap in resources for under-resourced languages.

Findings

Successfully trained a 51.65M-parameter Romanian language model from scratch.

Achieved a compact 26.45M-parameter model through quantization and pruning.

Generated three million Romanian synthetic fables with controlled prompting.

Abstract

Recent advances in synthetic data generation have shown that compact language models can be trained effectively when the underlying corpus is structurally controlled and linguistically coherent. However, for morphologically rich and computationally under-resourced languages such as Romanian, there is still no openly documented, end-to-end pipeline that unifies tokenizer design, preprocessing, pretraining, compression, evaluation, and large-scale synthetic data generation in a reproducible framework. Building on TF1, a three-million-story English fable dataset, and TF2, which extends TF1 through high-quality Romanian translations, we introduce TF3-RO, a Romanian-centric language modeling pipeline spanning tokenizer training, from-scratch model development, and Romanian-native dataset generation. TF3-RO constructs Romanian-specific BPE and Unigram tokenizers from a linguistically informed corpus to mitigate token inflation induced by Romanian morphology. Using long-sequence packed training, we pretrain a 51.65M-parameter LLaMA-style Transformer entirely from scratch. The model is subsequently optimized through quantization, structured pruning, and logit-based knowledge distillation, yielding a compact 26.45M-parameter student model with tied embeddings and strong deployment characteristics. Using this distilled model, TF3-RO generates three million Romanian-native synthetic fables via a controlled combinatorial prompting framework. Across all stages, the pipeline integrates a comprehensive evaluation suite combining intrinsic metrics, Romanian agreement probes, entity coherence, rule-based grammar checking, and LLM-based assessment. TF3-RO provides a reproducible and linguistically grounded framework for training compact Romanian language models and producing large-scale synthetic narrative corpora.