NeuroLoRA: Context-Aware Neuromodulation for Parameter-Efficient Multi-Task Adaptation

TL;DR

NeuroLoRA introduces a context-aware, neuromodulation-inspired Mixture-of-Experts framework for parameter-efficient multi-task adaptation of large language models, improving task decoupling and continual learning.

Contribution

It proposes a novel neuromodulation-based MoE approach with a contrastive orthogonality loss, enhancing multi-task adaptation and continual learning in LLMs.

Findings

Outperforms FlyLoRA and other baselines on MMLU, GSM8K, and ScienceQA.

Maintains parameter efficiency while improving task decoupling.

Effective in single-task, multi-task, and continual learning scenarios.

Abstract

Parameter-Efficient Fine-Tuning (PEFT) techniques, particularly Low-Rank Adaptation (LoRA), have become essential for adapting Large Language Models (LLMs) to downstream tasks. While the recent FlyLoRA framework successfully leverages bio-inspired sparse random projections to mitigate parameter interference, it relies on a static, magnitude-based routing mechanism that is agnostic to input context. In this paper, we propose NeuroLoRA, a novel Mixture-of-Experts (MoE) based LoRA framework inspired by biological neuromodulation -- the dynamic regulation of neuronal excitability based on context. NeuroLoRA retains the computational efficiency of frozen random projections while introducing a lightweight, learnable neuromodulation gate that contextually rescales the projection space prior to expert selection. We further propose a Contrastive Orthogonality Loss to explicitly enforce separation between expert subspaces, enhancing both task decoupling and continual learning capacity. Extensive experiments on MMLU, GSM8K, and ScienceQA demonstrate that NeuroLoRA consistently outperforms FlyLoRA and other strong baselines across single-task adaptation, multi-task model merging, and sequential continual learning scenarios, while maintaining comparable parameter efficiency.