stable-pretraining-v1: Foundation Model Research Made Simple

TL;DR

This paper introduces stable-pretraining, a flexible and efficient library built on PyTorch and Hugging Face, designed to simplify and accelerate foundation model research through modular utilities, comprehensive logging, and scalable experimentation.

Contribution

It presents a modular, extensible, and performance-optimized toolkit that unifies essential SSL utilities, enhancing flexibility, reproducibility, and speed in foundation model research.

Findings

Demonstrated ability to generate new research insights with minimal overhead.

Validated utility through depthwise representation probing and CLIP degradation analysis.

Enabled scalable experiments with comprehensive logging and monitoring.

Abstract

Foundation models and self-supervised learning (SSL) have become central to modern AI, yet research in this area remains hindered by complex codebases, redundant re-implementations, and the heavy engineering burden of scaling experiments. We present stable-pretraining, a modular, extensible, and performance-optimized library built on top of PyTorch, Lightning, Hugging Face, and TorchMetrics. Unlike prior toolkits focused narrowly on reproducing state-of-the-art results, stable-pretraining is designed for flexibility and iteration speed: it unifies essential SSL utilities--including probes, collapse detection metrics, augmentation pipelines, and extensible evaluation routines--within a coherent and reliable framework. A central design principle is logging everything, enabling fine-grained visibility into training dynamics that makes debugging, monitoring, and reproducibility seamless. We validate the library by demonstrating its ability to generate new research insights with minimal overhead, including depthwise representation probing and the analysis of CLIP degradation under synthetic data finetuning. By lowering barriers to entry while remaining scalable to large experiments, stable-pretraining aims to accelerate discovery and expand the possibilities of foundation model research.