MING-MOE: Enhancing Medical Multi-Task Learning in Large Language Models with Sparse Mixture of Low-Rank Adapter Experts

TL;DR

MING-MOE is a novel medical large language model that leverages a sparse mixture of low-rank adapters to handle diverse medical tasks efficiently without task-specific annotations, achieving state-of-the-art results.

Contribution

The paper introduces MING-MOE, a MOE-based model utilizing MoLoRA for efficient multi-task learning in medical NLP without requiring task-specific annotations.

Findings

Achieves SOTA on over 20 medical tasks

Improves inference efficiency

Handles diverse medical tasks without task-specific annotations

Abstract

Large language models like ChatGPT have shown substantial progress in natural language understanding and generation, proving valuable across various disciplines, including the medical field. Despite advancements, challenges persist due to the complexity and diversity inherent in medical tasks which often require multi-task learning capabilities. Previous approaches, although beneficial, fall short in real-world applications because they necessitate task-specific annotations at inference time, limiting broader generalization. This paper introduces MING-MOE, a novel Mixture-of-Expert~(MOE)-based medical large language model designed to manage diverse and complex medical tasks without requiring task-specific annotations, thus enhancing its usability across extensive datasets. MING-MOE employs a Mixture of Low-Rank Adaptation (MoLoRA) technique, allowing for efficient parameter usage by maintaining base model parameters static while adapting through a minimal set of trainable parameters. We demonstrate that MING-MOE achieves state-of-the-art (SOTA) performance on over 20 medical tasks, illustrating a significant improvement over existing models. This approach not only extends the capabilities of medical language models but also improves inference efficiency.