Active Domain Knowledge Acquisition with 100-Dollar Budget: Enhancing LLMs via Cost-Efficient, Expert-Involved Interaction in Sensitive Domains

TL;DR

This paper introduces PU-ADKA, a cost-efficient framework for enhancing large language models in specialized domains by actively involving experts within a fixed budget, validated through simulations and real-world deployment.

Contribution

The paper presents a novel active knowledge acquisition framework that optimally selects experts for cost-effective domain-specific LLM enhancement, along with a new benchmark dataset CKAD.

Findings

PU-ADKA improves LLM performance in specialized domains.

Effective expert selection reduces costs while maintaining quality.

Validated on PubMed data and real-world drug development team.

Abstract

Large Language Models (LLMs) have demonstrated an impressive level of general knowledge. However, they often struggle in highly specialized and cost-sensitive domains such as drug discovery and rare disease research due to the lack of expert knowledge. In this paper, we propose a novel framework (PU-ADKA) designed to efficiently enhance domain-specific LLMs by actively engaging domain experts within a fixed budget. Unlike traditional fine-tuning approaches, PU-ADKA selectively identifies and queries the most appropriate expert from a team, taking into account each expert's availability, knowledge boundaries, and consultation costs. We train PU-ADKA using simulations on PubMed data and validate it through both controlled expert interactions and real-world deployment with a drug development team, demonstrating its effectiveness in enhancing LLM performance in specialized domains under strict budget constraints. In addition to outlining our methodological innovations and experimental results, we introduce a new benchmark dataset, CKAD, for cost-effective LLM domain knowledge acquisition to foster further research in this challenging area.