PretrainZero: Reinforcement Active Pretraining

TL;DR

PretrainZero introduces a reinforcement active pretraining framework that enables large models to learn reasoning abilities from unlabeled data, significantly improving general reasoning without relying on domain-specific rewards.

Contribution

It proposes a novel reinforcement active pretraining method that enhances reasoning capabilities of large models using self-supervised RL on general corpora, bypassing the need for labeled data.

Findings

Improves reasoning benchmarks by 8.43 points on MMLU-Pro

Enhances general reasoning abilities through tackling complex masked spans

Enables pretrained models to serve as reasoning foundation models for downstream tasks

Abstract

Mimicking human behavior to actively learning from general experience and achieve artificial general intelligence has always been a human dream. Recent reinforcement learning (RL) based large-thinking models demonstrate impressive expert-level abilities, i.e., software and math, but still rely heavily on verifiable rewards in specific domains, placing a significant bottleneck to extend the performance boundary of general reasoning capabilities. In this work, we propose PretrainZero, a reinforcement active learning framework built on the pretraining corpus to extend RL from domain-specific post-training to general pretraining. PretrainZero features the following characteristics: 1) Active pretraining: inspired by the active learning ability of humans, PretrainZero learns a unified reasoning policy to actively identify reasonable and informative contents from pretraining corpus, and reason to predict these contents by RL. 2) Self-supervised learning: without any verifiable labels, pretrained reward models, or supervised fine-tuning, we directly pretrain reasoners from 3 to 30B base models on the general Wikipedia corpus using RL, significantly breaking the verification data-wall for general reasoning. 3) Verification scaling: by tackling increasingly challenging masked spans, PretrainZero substantially enhances the general reasoning abilities of pretrained base models. In reinforcement pretraining, PretrainZero improves Qwen3-4B-Base for 8.43, 5.96 and 10.60 on MMLU-Pro, SuperGPQA and math average benchmarks. In post-training, the pretrained models can also serve as reasoning foundation models for downstream RLVR tasks.