xGen-small Technical Report

TL;DR

xGen-small is a new family of Transformer models designed for long-context tasks, combining innovative data curation, multi-stage pre-training, and targeted fine-tuning to achieve strong performance in math, coding, and long-context benchmarks.

Contribution

It introduces a comprehensive pipeline for training long-context Transformer models, including data curation, multi-stage pre-training, and targeted post-training, which is novel in this domain.

Findings

Strong performance in math and coding tasks

Excels at long context benchmarks

Effective long-context modeling up to 128k tokens

Abstract

We introduce xGen-small, a family of 4B and 9B Transformer decoder models optimized for long-context applications. Our vertically integrated pipeline unites domain-balanced, frequency-aware data curation; multi-stage pre-training with quality annealing and length extension to 128k tokens; and targeted post-training via supervised fine-tuning, preference learning, and online reinforcement learning. xGen-small delivers strong performance across various tasks, especially in math and coding domains, while excelling at long context benchmarks.