Customizable Combination of Parameter-Efficient Modules for Multi-Task Learning

TL;DR

This paper presents C-Poly, a modular multi-task learning method that combines shared and task-specific skills with low-rank parameterization, improving generalization and sample efficiency across benchmarks.

Contribution

Introduces C-Poly, a novel customizable module combining shared and task-specific skills with low-rank techniques for multi-task learning.

Findings

C-Poly outperforms baseline methods on Super-NaturalInstructions and SuperGLUE.

Significantly improves sample efficiency in multi-task learning.

Effectively balances shared and task-specific knowledge.

Abstract

Modular and composable transfer learning is an emerging direction in the field of Parameter Efficient Fine-Tuning, as it enables neural networks to better organize various aspects of knowledge, leading to improved cross-task generalization. In this paper, we introduce a novel approach Customized Polytropon C-Poly that combines task-common skills and task-specific skills, while the skill parameters being highly parameterized using low-rank techniques. Each task is associated with a customizable number of exclusive specialized skills and also benefits from skills shared with peer tasks. A skill assignment matrix is jointly learned. To evaluate our approach, we conducted extensive experiments on the Super-NaturalInstructions and the SuperGLUE benchmarks. Our findings demonstrate that C-Poly outperforms fully-shared, task-specific, and skill-indistinguishable baselines, significantly enhancing the sample efficiency in multi-task learning scenarios.