propella-1: Multi-Property Document Annotation for LLM Data Curation at Scale

TL;DR

Propella-1 introduces a family of multilingual LLMs that provide multi-property, structured annotations for documents, enabling nuanced data curation and analysis at scale, surpassing single-score methods in interpretability and detail.

Contribution

The paper presents propella-1, a novel set of multilingual LLMs that generate detailed, multi-property document annotations, and releases a large dataset for improved data quality assessment.

Findings

Propella-4B outperforms larger models in annotation agreement.

Over three billion document annotations released for diverse datasets.

Reveals significant differences in dataset quality and content.

Abstract

Since FineWeb-Edu, data curation for LLM pretraining has predominantly relied on single scalar quality scores produced by small classifiers. A single score conflates multiple quality dimensions, prevents flexible filtering, and offers no interpretability. We introduce propella-1, a family of small multilingual LLMs (0.6B, 1.7B, 4B parameters) that annotate text documents across 18 properties organized into six categories: core content, classification, quality and value, audience and purpose, safety and compliance, and geographic relevance. The models support 57 languages and produce structured JSON annotations conforming to a predefined schema. Evaluated against a frontier commercial LLM as a reference annotator, the 4B model achieves higher agreement than much larger general-purpose models. We release propella-annotations, a dataset of over three billion document annotations covering major pretraining corpora including data from FineWeb-2, FinePDFs, HPLT 3.0, and Nemotron-CC. Using these annotations, we present a multi-dimensional compositional analysis of widely used pretraining datasets, revealing substantial differences in quality, reasoning depth, and content composition that single-score approaches cannot capture. All model weights and annotations are released under permissive, commercial-use licenses.