Structure-aware Domain Knowledge Injection for Large Language Models

TL;DR

This paper presents StructTuning, a novel approach that efficiently transforms large language models into domain experts by leveraging structured knowledge, reducing training data needs while maintaining high performance.

Contribution

The paper introduces StructTuning, a two-stage, structure-aware method for domain knowledge injection into LLMs, significantly improving data efficiency and model performance.

Findings

Achieves full knowledge injection performance with only 5% of training data.

Outperforms existing knowledge injection methods on LongBench and MMedBench datasets.

Demonstrates scalability across different model sizes and training corpus scales.

Abstract

This paper introduces a pioneering methodology, termed StructTuning, to efficiently transform foundation Large Language Models (LLMs) into domain specialists. It significantly reduces the training corpus needs to a mere 5% while achieving an impressive 100% of traditional knowledge injection performance. Motivated by structured human education, we propose a novel two-stage strategy for knowledge injection and alignment: Structure-aware Continual Pre-Training (SCPT) and Structure-aware Supervised Fine-Tuning (SSFT). In the SCPT phase, we automatically extract the domain knowledge taxonomy and reorganize the training corpora, enabling LLMs to effectively link textual segments to targeted knowledge points within the taxonomy. In the SSFT phase, we explicitly prompt models to elucidate the underlying knowledge structure in their outputs, leveraging the structured domain insight to address practical problems. Our ultimate method was extensively evaluated across model architectures and scales on LongBench and MMedBench datasets, demonstrating superior performance against other knowledge injection methods. We also explored our method's scalability across different training corpus sizes, laying the foundation to enhance domain-specific LLMs with better data utilization.