Improving Natural Language Understanding for LLMs via Large-Scale Instruction Synthesis

TL;DR

This paper introduces Hum, a large-scale synthetic instruction dataset for NLU tasks, created through human-LLM collaboration, significantly improving LLMs' NLU performance without harming general capabilities.

Contribution

The paper presents Hum, a diverse, high-quality instruction corpus for NLU, and a collaborative synthesis method to enhance LLMs' NLU abilities.

Findings

Hum improves NLU performance by 3.1% on average

Diverse instruction data maintains overall model capabilities

Collaborative synthesis enriches instruction quality and variety

Abstract

High-quality, large-scale instructions are crucial for aligning large language models (LLMs), however, there is a severe shortage of instruction in the field of natural language understanding (NLU). Previous works on constructing NLU instructions mainly focus on information extraction (IE), neglecting tasks such as machine reading comprehension, question answering, and text classification. Furthermore, the lack of diversity in the data has led to a decreased generalization ability of trained LLMs in other NLU tasks and a noticeable decline in the fundamental model's general capabilities. To address this issue, we propose Hum, a large-scale, high-quality synthetic instruction corpus for NLU tasks, designed to enhance the NLU capabilities of LLMs. Specifically, Hum includes IE (either close IE or open IE), machine reading comprehension, text classification, and instruction generalist tasks, thereby enriching task diversity. Additionally, we introduce a human-LLMs collaborative mechanism to synthesize instructions, which enriches instruction diversity by incorporating guidelines, preference rules, and format variants. We conduct extensive experiments on 5 NLU tasks and 28 general capability evaluation datasets for LLMs. Experimental results show that Hum enhances the NLU capabilities of six LLMs by an average of 3.1\%, with no significant decline observed in other general capabilities.