Mangosteen: An Open Thai Corpus for Language Model Pretraining

TL;DR

Mangosteen is a comprehensive, openly available 47-billion-token Thai corpus created with a specialized pipeline, significantly improving Thai language model pretraining and setting a reproducible standard for future regional NLP research.

Contribution

The paper introduces Mangosteen, a large, transparent Thai corpus built with a Thai-adapted pipeline, and demonstrates its effectiveness in enhancing Thai language models.

Findings

Pipeline reduces web data from 202M to 25M documents.

Pretraining on Mangosteen improves Thai NLP benchmark scores.

Open release of data and tools supports reproducibility.

Abstract

Pre-training data shapes a language model's quality, but raw web text is noisy and demands careful cleaning. Existing large-scale corpora rely on English-centric or language-agnostic pipelines whose heuristics do not capture Thai script or cultural nuances, leaving risky material such as gambling content untreated. Prior Thai-specific efforts customize pipelines or build new ones, yet seldom release their data or document design choices, hindering reproducibility and raising the question of how to construct a transparent, high-quality Thai corpus. We introduce Mangosteen: a 47 billion-token Thai corpus built through a Thai-adapted Dolma pipeline that includes custom rule-based language ID, revised C4/Gopher quality filters, and Thai-trained content filters, plus curated non-web sources such as Wikipedia, Royal Gazette texts, OCR-extracted books, and CC-licensed YouTube subtitles. Systematic ablations using GPT-2 show the pipeline trims CommonCrawl from 202M to 25M documents while raising SEA-HELM NLG from 3 to 11; an 8B-parameter SEA-LION model continually pre-trained on Mangosteen then surpasses SEA-LION-v3 and Llama-3.1 by about four points on Thai benchmarks. We release the full pipeline code, cleaning manifests, corpus snapshot, and all checkpoints, providing a fully reproducible foundation for future Thai and regional LLM research.