Parameter-efficient Multi-Task and Multi-Domain Learning using Factorized Tensor Networks

TL;DR

This paper introduces a factorized tensor network (FTN) that enables multi-task and multi-domain learning with high accuracy and minimal additional parameters, avoiding catastrophic forgetting and improving efficiency.

Contribution

The paper proposes a novel FTN approach that incorporates low-rank tensor factors into a shared frozen network for efficient multi-task and multi-domain learning.

Findings

FTN achieves comparable accuracy to independent models.

FTN uses significantly fewer task-specific parameters.

Experiments validate FTN across various architectures and datasets.

Abstract

Multi-task and multi-domain learning methods seek to learn multiple tasks/domains, jointly or one after another, using a single unified network. The primary challenge and opportunity lie in leveraging shared information across these tasks and domains to enhance the efficiency of the unified network. The efficiency can be in terms of accuracy, storage cost, computation, or sample complexity. In this paper, we introduce a factorized tensor network (FTN) designed to achieve accuracy comparable to that of independent single-task or single-domain networks, while introducing a minimal number of additional parameters. The FTN approach entails incorporating task- or domain-specific low-rank tensor factors into a shared frozen network derived from a source model. This strategy allows for adaptation to numerous target domains and tasks without encountering catastrophic forgetting. Furthermore, FTN requires a significantly smaller number of task-specific parameters compared to existing methods. We performed experiments on widely used multi-domain and multi-task datasets. We show the experiments on convolutional-based architecture with different backbones and on transformer-based architecture. Our findings indicate that FTN attains similar accuracy as single-task or single-domain methods while using only a fraction of additional parameters per task. The code is available at https://doi.org/10.24433/CO.7519211.v2.