Qwen Technical Report

TL;DR

Qwen is a comprehensive series of large language models including base, chat, coding, and math variants, demonstrating strong performance and advanced capabilities across various NLP tasks and applications.

Contribution

Introduction of the Qwen series, including models with human-aligned fine-tuning and specialized variants for coding and mathematics, advancing open-source LLM capabilities.

Findings

Base models outperform many open-source models on downstream tasks.

Chat models with RLHF show high competitiveness and advanced tool-use.

Specialized models like Code-Qwen and Math-Qwen improve performance in their domains.

Abstract

Large language models (LLMs) have revolutionized the field of artificial intelligence, enabling natural language processing tasks that were previously thought to be exclusive to humans. In this work, we introduce Qwen, the first installment of our large language model series. Qwen is a comprehensive language model series that encompasses distinct models with varying parameter counts. It includes Qwen, the base pretrained language models, and Qwen-Chat, the chat models finetuned with human alignment techniques. The base language models consistently demonstrate superior performance across a multitude of downstream tasks, and the chat models, particularly those trained using Reinforcement Learning from Human Feedback (RLHF), are highly competitive. The chat models possess advanced tool-use and planning capabilities for creating agent applications, showcasing impressive performance even when compared to bigger models on complex tasks like utilizing a code interpreter. Furthermore, we have developed coding-specialized models, Code-Qwen and Code-Qwen-Chat, as well as mathematics-focused models, Math-Qwen-Chat, which are built upon base language models. These models demonstrate significantly improved performance in comparison with open-source models, and slightly fall behind the proprietary models.