Separating Shared and Domain-Specific LoRAs for Multi-Domain Learning

TL;DR

This paper introduces a method to separate shared and domain-specific LoRA adapters into different subspaces for multi-domain learning, improving the capture of domain-specific information.

Contribution

It proposes a novel approach to explicitly separate shared and domain-specific LoRA components into different subspaces, enhancing multi-domain learning effectiveness.

Findings

Effective in action recognition datasets

Analyzes LoRA weight dimensions

Shows cases of improved domain-specific capture

Abstract

Existing architectures of multi-domain learning have two types of adapters: shared LoRA for all domains and domain-specific LoRA for each particular domain. However, it remains unclear whether this structure effectively captures domain-specific information. In this paper, we propose a method that ensures that shared and domain-specific LoRAs exist in different subspaces; specifically, the column and left null subspaces of the pre-trained weights. We apply the proposed method to action recognition with three datasets (UCF101, Kinetics400, and HMDB51) and demonstrate its effectiveness in some cases along with the analysis of the dimensions of LoRA weights.