BLADE: Enhancing Black-box Large Language Models with Small Domain-Specific Models

TL;DR

BLADE is a framework that enhances black-box large language models with small, domain-specific models to improve performance in specialized fields like legal and medical domains, without extensive retraining.

Contribution

BLADE introduces a novel method combining a small domain-specific LM with a general LLM via Bayesian optimization, offering a cost-effective way to adapt LLMs for vertical domains.

Findings

Outperforms existing domain adaptation methods on legal and medical benchmarks.

Significantly improves domain-specific task accuracy with minimal additional training.

Demonstrates cost efficiency compared to continuous pre-training or retrieval augmentation.

Abstract

Large Language Models (LLMs) like ChatGPT and GPT-4 are versatile and capable of addressing a diverse range of tasks. However, general LLMs, which are developed on open-domain data, may lack the domain-specific knowledge essential for tasks in vertical domains, such as legal, medical, etc. To address this issue, previous approaches either conduct continuous pre-training with domain-specific data or employ retrieval augmentation to support general LLMs. Unfortunately, these strategies are either cost-intensive or unreliable in practical applications. To this end, we present a novel framework named BLADE, which enhances Black-box LArge language models with small Domain-spEcific models. BLADE consists of a black-box LLM and a small domain-specific LM. The small LM preserves domain-specific knowledge and offers specialized insights, while the general LLM contributes robust language comprehension and reasoning capabilities. Specifically, our method involves three steps: 1) pre-training the small LM with domain-specific data, 2) fine-tuning this model using knowledge instruction data, and 3) joint Bayesian optimization of the general LLM and the small LM. Extensive experiments conducted on public legal and medical benchmarks reveal that BLADE significantly outperforms existing approaches. This shows the potential of BLADE as an effective and cost-efficient solution in adapting general LLMs for vertical domains.