DELIA: Diversity-Enhanced Learning for Instruction Adaptation in Large Language Models

TL;DR

This paper introduces DELIA, a novel data synthesis method that enhances instruction tuning in large language models by transforming biased features into more ideal, semantic-rich features through diverse data, leading to improved performance.

Contribution

DELIA is a new data synthesis approach that leverages diverse data to improve instruction adaptation in LLMs without prior knowledge of ideal features.

Findings

DELIA outperforms standard instruction tuning by 17-33% in translation quality.

DELIA improves accuracy by 36.1% in formatted text generation.

DELIA uniquely aligns new token representations with their prior semantics.

Abstract

Although instruction tuning is widely used to adjust behavior in Large Language Models (LLMs), extensive empirical evidence and research indicates that it is primarily a process where the model fits to specific task formats, rather than acquiring new knowledge or capabilities. We propose that this limitation stems from biased features learned during instruction tuning, which differ from ideal task-specfic features, leading to learn less underlying semantics in downstream tasks. However, ideal features are unknown and incalculable, constraining past work to rely on prior knowledge to assist reasoning or training, which limits LLMs' capabilities to the developers' abilities, rather than data-driven scalable learning. In our paper, through our novel data synthesis method, DELIA (Diversity-Enhanced Learning for Instruction Adaptation), we leverage the buffering effect of extensive diverse data in LLMs training to transform biased features in instruction tuning into approximations of ideal features, without explicit prior ideal features. Experiments show DELIA's better performance compared to common instruction tuning and other baselines. It outperforms common instruction tuning by 17.07%-33.41% on Icelandic-English translation bleurt score (WMT-21 dataset, gemma-7b-it) and improves accuracy by 36.1% on formatted text generation (Llama2-7b-chat). Notably, among knowledge injection methods we've known, DELIA uniquely align the internal representations of new special tokens with their prior semantics.