Self-Improving Pretraining: using post-trained models to pretrain better models

TL;DR

This paper proposes a novel pretraining approach that integrates post-trained models to improve early-stage learning of safety, factuality, and reasoning in large language models.

Contribution

It introduces a method to incorporate behaviors learned in post-training into pretraining, enabling earlier reinforcement of desirable model capabilities.

Findings

Significant improvements in safety, factuality, and reasoning abilities.

Enhanced overall generation quality.

Effective use of existing strong models for rewriting data and judging rollouts.

Abstract

Large language models are classically trained in stages: pretraining on raw text followed by post-training for instruction following and reasoning. However, this separation creates a fundamental limitation: many desirable behaviors such as safety, factuality, overall generation quality, and reasoning ability are only added at a late stage, even though the patterns learned earlier strongly shape a model's capabilities. To tackle this issue, we introduce a new way to pretrain and mid-train models that incorporates these behaviors earlier. We utilize an existing strong, post-trained model to both rewrite pretraining data and to judge policy model rollouts, thus using reinforcement earlier in training. In our experiments, we show this can give strong gains in quality, safety, factuality and reasoning.