An Engineering Journey Training Large Language Models at Scale on Alps: The Apertus Experience

TL;DR

This paper describes the engineering process of training a large, multilingual AI model on Europe's Alps supercomputer, highlighting infrastructure challenges and solutions for academic-scale LLM training.

Contribution

It presents the first successful training of a 70B parameter LLM in academia using HPC infrastructure, detailing technical challenges and platform evolution.

Findings

Achieved training of a 70B parameter LLM on Alps supercomputer.

Identified and addressed HPC infrastructure challenges for AI training.

Developed a resilient, scalable ML platform for ongoing model fine-tuning.

Abstract

Large Language Models (LLMs) have surged as a transformative technology for science and society, prompting governments worldwide to pursue sovereign AI capabilities that ensure data compliance and cultural representation. However, the associated capital costs and engineering complexity required to train these models have largely restricted such capabilities to the private sector, leaving a significant gap for public institutions. This paper details the engineering journey behind training Apertus, a fully open multilingual foundation model, on the Alps supercomputer. Representing a first-of-its-kind achievement for academia at the 70B parameter scale, we successfully deployed a massive pre-training campaign on one of Europe's largest systems for open science, powered by NVIDIA GH200 Grace Hopper Superchips. We detail the challenges encountered in readying HPC infrastructure for training AI models, from overcoming storage bottlenecks to stabilizing large-scale interconnects, and the lessons learned in transforming a supercomputer into a resilient software-defined Machine Learning Platform. Finally, we discuss the post-training requirements and evolution of our Machine Learning platform, outlining how this initial release lays the groundwork for a sustained, iterative operational capability, in particular for fine tuning foundation models, that extends well beyond a single model training run.