DoMIX: An Efficient Framework for Exploiting Domain Knowledge in Fine-Tuning

TL;DR

DoMIX introduces a parameter-efficient framework using LoRA modules for robust, efficient, and domain-specific fine-tuning of large language models, overcoming limitations of previous continual DAP methods.

Contribution

The paper presents DoMIX, a novel, efficient, and robust framework leveraging LoRA modules for domain-adaptive pre-training that addresses existing challenges in continual DAP.

Findings

Efficient domain-adaptive pre-training with reduced computational cost.

Robustness to domain order in continual DAP.

Ability to produce tailored models for specific tasks.

Abstract

Domain-Adaptive Pre-training (DAP) has recently gained attention for its effectiveness in fine-tuning pre-trained models. Building on this, continual DAP has been explored to develop pre-trained models capable of incrementally incorporating different domain datasets. However, existing continual DAP methods face several limitations: (1) high computational cost and GPU memory usage during training; (2) sensitivity to incremental data order; and (3) providing a single, generalized model for all end tasks, which contradicts the essence of DAP. In this paper, we propose DoMIX, a novel approach that addresses these challenges by leveraging LoRA modules, a representative parameter-efficient fine-tuning (PEFT) method. Our approach enables efficient and parallel domain-adaptive pre-training that is robust to domain order and effectively utilizes accumulated knowledge to provide tailored pre-trained models for specific tasks. We also demonstrate that our method can be extended beyond the DAP setting to standard LLM fine-tuning scenarios. Code is available at https://github.com/dohoonkim-ai/DoMIX.