Next Concept Prediction in Discrete Latent Space Leads to Stronger Language Models

TL;DR

This paper introduces Next Concept Prediction (NCP), a new pretraining method for language models that predicts multi-token concepts, resulting in stronger models with improved performance across benchmarks.

Contribution

The paper presents NCP, a novel pretraining paradigm that enhances language models by predicting discrete multi-token concepts, outperforming traditional token-level training.

Findings

NCP improves performance on 13 benchmarks.

NCP enhances models trained from scratch and via continual pretraining.

ConceptLM achieves better results with larger scales and datasets.

Abstract

We propose Next Concept Prediction (NCP), a generative pretraining paradigm built on top of Next Token Prediction (NTP). NCP predicts discrete concepts that span multiple tokens, thereby forming a more challenging pretraining objective. Our model, ConceptLM, quantizes hidden states using Vector Quantization and constructs a concept vocabulary. It leverages both NCP and NTP to drive parameter updates and generates a concept to guide the generation of the following tokens. We train ConceptLM from scratch at scales ranging from 70M to 1.5B parameters with up to 300B training data, including Pythia and GPT-2 backbones. Results on 13 benchmarks show that NCP yields consistent performance gains over traditional token-level models. Furthermore, continual pretraining experiments on an 8B-parameter Llama model indicate that NCP can further improve an NTP-trained model. Our analysis suggests that NCP leads to more powerful language models by introducing a harder pretraining task, providing a promising path toward better language modeling.