When MOE Meets LLMs: Parameter Efficient Fine-tuning for Multi-task Medical Applications

TL;DR

This paper introduces MOELoRA, a novel parameter-efficient fine-tuning framework combining mixture-of-experts and low-rank adaptation for multi-task medical applications, addressing task diversity and computational efficiency.

Contribution

It proposes a unified framework that integrates MOE and LoRA for efficient multi-task fine-tuning of LLMs in medical domains, with a task-motivated gating mechanism.

Findings

MOELoRA outperforms existing methods on multi-task medical datasets.

The framework reduces parameter count and computational cost.

It effectively handles task diversity in medical applications.

Abstract

The recent surge in Large Language Models (LLMs) has garnered significant attention across numerous fields. Fine-tuning is often required to fit general LLMs for a specific domain, like the web-based healthcare system. However, two problems arise during fine-tuning LLMs for medical applications. One is the task variety problem, which involves distinct tasks in real-world medical scenarios. The variety often leads to sub-optimal fine-tuning for data imbalance and seesaw problems. Besides, the large amount of parameters in LLMs leads to huge time and computation consumption by fine-tuning. To address these two problems, we propose a novel parameter efficient fine-tuning framework for multi-task medical applications, dubbed as MOELoRA. The designed framework aims to absorb both the benefits of mixture-of-expert (MOE) for multi-task learning and low-rank adaptation (LoRA) for parameter efficient fine-tuning. For unifying MOE and LoRA, we devise multiple experts as the trainable parameters, where each expert consists of a pair of low-rank matrices to retain the small size of trainable parameters. Then, a task-motivated gate function for all MOELoRA layers is proposed, which can control the contributions of each expert and produce distinct parameters for various tasks. We conduct experiments on a multi-task medical dataset, indicating MOELoRA outperforms the existing parameter efficient fine-tuning methods. The code is available online.