Training Foundation Models on a Full-Stack AMD Platform: Compute, Networking, and System Design

TL;DR

This paper presents the first large-scale mixture-of-experts pretraining on AMD hardware, providing detailed system benchmarks, model design guidance, and a new 760M parameter MoE model called ZAYA1 that performs competitively on various benchmarks.

Contribution

It offers comprehensive system characterization, microbenchmark data, and MI300X-aware transformer sizing rules, along with the development of the ZAYA1 MoE model for large-scale pretraining on AMD hardware.

Findings

ZAYA1 achieves performance comparable to leading models.

AMD hardware and software stack are mature for large-scale training.

Detailed system benchmarks inform model and system design choices.

Abstract

We report on the first large-scale mixture-of-experts (MoE) pretraining study on pure AMD hardware, utilizing both MI300X GPUs and Pollara networking. We distill practical guidance for both systems and model design. On the systems side, we deliver a comprehensive cluster and networking characterization: microbenchmarks for all core collectives (all-reduce, reduce-scatter, all-gather, broadcast) across message sizes and GPU counts over Pollara. To our knowledge, this is the first at this scale. We further provide MI300X microbenchmarks on kernel sizing and memory bandwidth to inform model design. On the modeling side, we introduce and apply MI300X-aware transformer sizing rules for attention and MLP blocks and justify MoE widths that jointly optimize training throughput and inference latency. We describe our training stack in depth, including often-ignored utilities such as fault-tolerance and checkpoint-reshaping, as well as detailed information on our training recipe. We also provide a preview of our model architecture and base model - ZAYA1 (760M active, 8.3B total parameters MoE, available at https://huggingface.co/Zyphra/ZAYA1-base) - which will be further improved upon in forthcoming papers. ZAYA1-base achieves performance comparable to leading base models such as Qwen3-4B and Gemma3-12B at its scale and larger, and outperforms models including Llama-3-8B and OLMoE across reasoning, mathematics, and coding benchmarks. Together, these results demonstrate that the AMD hardware, network, and software stack are mature and optimized enough for competitive large-scale pretraining.