Super Apriel: One Checkpoint, Many Speeds

TL;DR

Super Apriel introduces a versatile 15B-parameter supernet with multiple decoder layer mixers, enabling flexible speed-quality tradeoffs and speculative decoding from a single checkpoint, with extensive configuration and optimization tools.

Contribution

It presents a multi-mixer supernet allowing dynamic speed-quality tradeoffs, surrogate-based configuration prediction, and comprehensive training and deployment tools.

Findings

All-FA preset matches the teacher on benchmarks.

Hybrid presets achieve 2.9x to 10.7x throughput with high quality retention.

Configuration stability varies with model size and training progress.

Abstract

We release Super Apriel, a 15B-parameter supernet in which every decoder layer provides four trained mixer choices -- Full Attention (FA), Sliding Window Attention (SWA), Kimi Delta Attention (KDA), and Gated DeltaNet (GDN). A placement selects one mixer per layer; placements can be switched between requests at serving time without reloading weights, enabling multiple speed presets from a single checkpoint. The shared checkpoint also enables speculative decoding without a separate draft model. The all-FA preset matches the Apriel 1.6 teacher on all reported benchmarks; recommended hybrid presets span $2.9\times$ to $10.7\times$ decode throughput at 96% to 77% quality retention, with throughput advantages that compound at longer context lengths. With four mixer types across 48 layers, the configuration space is vast. A surrogate that predicts placement quality from the per-layer mixer assignment makes the speed-quality landscape tractable and identifies the best tradeoffs at each speed level. We investigate whether the best configurations at each speed level can be identified early in training or only after convergence. Rankings stabilize quickly at 0.5B scale, but the most efficient configurations exhibit higher instability at 15B, cautioning against extrapolation from smaller models. Super Apriel is trained by stochastic distillation from a frozen Apriel 1.6 teacher, followed by supervised fine-tuning. We release the supernet weights, Fast-LLM training code, vLLM serving code, and a placement optimization toolkit.