ICM-Fusion: In-Context Meta-Optimized LoRA Fusion for Multi-Task Adaptation

TL;DR

ICM-Fusion introduces a meta-learning framework with in-context adaptation for multi-task LoRA fusion, effectively balancing conflicting task directions and reducing catastrophic forgetting, thus improving multi-task generalization especially in few-shot scenarios.

Contribution

The paper proposes ICM-Fusion, a novel meta-learning based framework that dynamically balances task conflicts and reconstructs fused LoRA models, advancing multi-task adaptation in pre-trained models.

Findings

ICM-Fusion reduces multi-tasking loss significantly.

It achieves task enhancement in few-shot scenarios.

The method is effective across visual and linguistic tasks.

Abstract

Enabling multi-task adaptation in pre-trained Low-Rank Adaptation (LoRA) models is crucial for enhancing their generalization capabilities. Most existing pre-trained LoRA fusion methods decompose weight matrices, sharing similar parameters while merging divergent ones. However, this paradigm inevitably induces inter-weight conflicts and leads to catastrophic domain forgetting. While incremental learning enables adaptation to multiple tasks, it struggles to achieve generalization in few-shot scenarios. Consequently, when the weight data follows a long-tailed distribution, it can lead to forgetting in the fused weights. To address this issue, we propose In-Context Meta LoRA Fusion (ICM-Fusion), a novel framework that synergizes meta-learning with in-context adaptation. The key innovation lies in our task vector arithmetic, which dynamically balances conflicting optimization directions across domains through learned manifold projections. ICM-Fusion obtains the optimal task vector orientation for the fused model in the latent space by adjusting the orientation of the task vectors. Subsequently, the fused LoRA is reconstructed by a self-designed Fusion VAE (F-VAE) to realize multi-task LoRA generation. We have conducted extensive experiments on visual and linguistic tasks, and the experimental results demonstrate that ICM-Fusion can be adapted to a wide range of architectural models and applied to various tasks. Compared to the current pre-trained LoRA fusion method, ICM-Fusion fused LoRA can significantly reduce the multi-tasking loss and can even achieve task enhancement in few-shot scenarios.