Aurora:Activating Chinese chat capability for Mixtral-8x7B sparse Mixture-of-Experts through Instruction-Tuning

TL;DR

This paper introduces Aurora, a Chinese conversational model based on Mixtral-8x7B, enhanced through instruction fine-tuning with Chinese datasets, demonstrating improved zero-shot performance on benchmark tests.

Contribution

It is the first to apply instruction fine-tuning to a sparse Mixture-of-Experts model for Chinese conversational capabilities.

Findings

Aurora outperforms baseline models on C-Eval, MMLU, and CMMLU benchmarks.

Instruction fine-tuning significantly improves Chinese conversational abilities.

The approach demonstrates the effectiveness of instruction tuning on sparse expert models.

Abstract

Existing research has demonstrated that refining large language models (LLMs) through the utilization of machine-generated instruction-following data empowers these models to exhibit impressive zero-shot capabilities for novel tasks, without requiring human-authored instructions. In this paper, we systematically investigate, preprocess, and integrate three Chinese instruction-following datasets with the aim of enhancing the Chinese conversational capabilities of Mixtral-8x7B sparse Mixture-of-Experts model. Through instruction fine-tuning on this carefully processed dataset, we successfully construct the Mixtral-8x7B sparse Mixture-of-Experts model named "Aurora." To assess the performance of Aurora, we utilize three widely recognized benchmark tests: C-Eval, MMLU, and CMMLU. Empirical studies validate the effectiveness of instruction fine-tuning applied to Mixtral-8x7B sparse Mixture-of-Experts model. This work is pioneering in the execution of instruction fine-tuning on a sparse expert-mixed model, marking a significant breakthrough in enhancing the capabilities of this model architecture. Our code, data and model are publicly available at https://github.com/WangRongsheng/Aurora