360Zhinao Technical Report

TL;DR

This paper introduces 360Zhinao, a 7-billion parameter language model with flexible context lengths up to 360K tokens, achieved through careful data preparation, training, and alignment strategies, demonstrating competitive performance.

Contribution

The paper presents a comprehensive development of 360Zhinao, including model training, data strategies, and extended context capabilities, with minimal model size ablation environment.

Findings

Achieved 360K context length extension for 360Zhinao-7B.

Trained on 3.4T tokens with refined data filtering.

Demonstrated competitive performance among similar-sized models.

Abstract

We present 360Zhinao models with 7B parameter size and context lengths spanning 4K, 32K and 360K, all available at https://github.com/Qihoo360/360zhinao. For rapid development in pretraining, we establish a stable and sensitive ablation environment to evaluate and compare experiment runs with minimal model size. Under such guidance, we perfect our data cleaning and composition strategies to pretrain $\texttt{360Zhinao-7B-Base}$ on 3.4T tokens. We also mainly emphasize data during alignment, where we strive to balance quantity and quality with filtering and reformatting. With tailored data, 360Zhinao-7B's context window is easily extended to 32K and 360K. RMs and RLHF are trained following SFT and credibly applied to specific tasks. All together these contributions lead to 360Zhinao-7B's competitive performance among models of similar size.