ThanoRA: Task Heterogeneity-Aware Multi-Task Low-Rank Adaptation

TL;DR

ThanoRA introduces a task heterogeneity-aware multi-task low-rank adaptation framework that improves multi-task learning efficiency and performance without extra inference costs by tailoring subspaces and preserving diversity.

Contribution

It proposes a novel method for multi-task adaptation that allocates task-specific subspaces based on heterogeneity and enforces diversity, outperforming existing approaches without additional inference overhead.

Findings

Outperforms strong baselines across multimodal and text benchmarks.

Surpasses separate task-specific fine-tuning in performance.

No additional inference overhead or structural modifications required.

Abstract

Low-Rank Adaptation (LoRA) is widely adopted for downstream fine-tuning of foundation models due to its efficiency and zero additional inference cost. Many real-world applications require foundation models to specialize in several specific tasks simultaneously, motivating the need for efficient multi-task downstream adaptation. To address this need, existing studies have primarily explored two directions: Model Merging with LoRA, which shows advantages in training-free scenarios but still lags behind multi-task training in overall performance; and MoE-based LoRA approaches, which improve multi-task learning performance but introduce routers that hinder the mergeability of LoRA parameters and incur considerable inference overhead, thereby limiting real-world deployment practicality. To this end, we propose ThanoRA, a Task Heterogeneity-Aware Multi-Task Low-Rank Adaptation framework that enables effective, efficient and unified multi-task downstream adaptation without introducing additional structure. ThanoRA performs multi-task learning by tailoring subspace allocation at initialization and enforcing diversity preservation throughout training: it allocates varying dimensions to construct task-specific low-rank subspaces driven by inter-task heterogeneity, enabling fine-grained knowledge injection, while diversity-preserving regularization mitigates task interference and subspace collapse, thereby fully exploiting the low-rank capacity. Extensive experiments across multimodal and text-only benchmarks under varying multi-task mixtures demonstrate that ThanoRA consistently outperforms strong baselines, surpassing even separate task-specific fine-tuning, while introducing no additional structures or inference overhead. Our code will be publicly available at: https://github.com/LiangJian24/ThanoRA.