Nyonic Technical Report

TL;DR

This report presents Wonton 7B, a multilingual language model with innovative training techniques and architecture enhancements, achieving competitive benchmark performance and emphasizing robustness and adaptability.

Contribution

Introduces a new training framework with an Online Data Scheduler and advanced architecture features for improved multilingual model performance.

Findings

Wonton 7B demonstrates competitive multilingual benchmark results.

The novel training framework enhances efficiency and stability.

State-of-the-art architectural techniques improve model robustness.

Abstract

This report details the development and key achievements of our latest language model designed for custom large language models. The advancements introduced include a novel Online Data Scheduler that supports flexible training data adjustments and curriculum learning. The model's architecture is fortified with state-of-the-art techniques such as Rotary Positional Embeddings, QK-LayerNorm, and a specially crafted multilingual tokenizer to enhance stability and performance. Moreover, our robust training framework incorporates advanced monitoring and rapid recovery features to ensure optimal efficiency. Our Wonton 7B model has demonstrated competitive performance on a range of multilingual and English benchmarks. Future developments will prioritize narrowing the performance gap with more extensively trained models, thereby enhancing the model's real-world efficacy and adaptability.GitHub: \url{https://github.com/nyonicai/nyonic-public}