Supernova: Achieving More with Less in Transformer Architectures

TL;DR

Supernova is a highly efficient 650M-parameter transformer that uses innovative architecture and tokenization to match larger models' performance with fewer parameters and training data.

Contribution

The paper introduces Supernova, a transformer architecture with novel design choices and a custom tokenizer that achieve high performance with fewer parameters and less training data.

Findings

Achieves 90% of 1B-parameter model performance

Uses 35% fewer parameters than larger models

Requires only 100B training tokens

Abstract

We present Supernova, a 650M-parameter decoder-only transformer that demonstrates how careful architectural design and tokenization innovation can achieve the performance of larger models while maintaining computational efficiency. Our architecture combines Rotary Positional Embeddings (RoPE), Grouped Query Attention (GQA) with a 3:1 compression ratio, RMSNorm for computational efficiency, and SwiGLU activation functions. A critical innovation is our custom 128,000-vocabulary byte-level BPE tokenizer, which achieves state-of-the-art compression performance. Through detailed analysis, we show that Supernova achieves 90% of the performance of 1B-parameter models while using 35% fewer parameters and requiring only 100B training tokens--an order of magnitude less than competing models. Our findings challenge the prevailing scaling paradigm, demonstrating that architectural efficiency and tokenization quality can compensate for reduced parameter counts.