Fox-1: Open Small Language Model for Cloud and Edge

TL;DR

Fox-1 introduces a series of small, efficient language models with a novel training curriculum and architecture enhancements, achieving competitive performance and accessibility for open-source communities.

Contribution

The paper presents Fox-1, a new small language model series with a unique 3-stage data curriculum and architectural improvements like Grouped Query Attention.

Findings

Achieves comparable or better performance than larger models on benchmarks.

Introduces a novel 3-stage data curriculum for improved training efficiency.

Models are openly released under Apache 2.0 license.

Abstract

We present Fox-1, a series of small language models (SLMs) consisting of Fox-1-1.6B and Fox-1-1.6B-Instruct-v0.1. These models are pre-trained on 3 trillion tokens of web-scraped document data and fine-tuned with 5 billion tokens of instruction-following and multi-turn conversation data. Aiming to improve the pre-training efficiency, Fox-1-1.6B model introduces a novel 3-stage data curriculum across all the training data with 2K-8K sequence length. In architecture design, Fox-1 features a deeper layer structure, an expanded vocabulary, and utilizes Grouped Query Attention (GQA), offering a performant and efficient architecture compared to other SLMs. Fox-1 achieves better or on-par performance in various benchmarks compared to StableLM-2-1.6B, Gemma-2B, Qwen1.5-1.8B, and OpenELM1.1B, with competitive inference speed and throughput. The model weights have been released under the Apache 2.0 license, where we aim to promote the democratization of LLMs and make them fully accessible to the whole open-source community.