Beyond Task Vectors: Selective Task Arithmetic Based on Importance Metrics
Tian Bowen, Lai Songning, Wu Jiemin, Shuai Zhihao, Ge Shiming, Yue, Yutao

TL;DR
This paper presents STA, a training-free framework that improves multi-task learning by selectively fusing task-specific parameters based on importance metrics, reducing hyperparameter tuning and enhancing task forgetting.
Contribution
STA introduces a novel, importance-based parameter fusion method for multi-task models, addressing hyperparameter reliance and enabling better task forgetting without additional training.
Findings
STA outperforms existing methods on multiple benchmarks.
It achieves more effective task forgetting with minimal performance loss.
The importance metric accurately identifies task-specific parameters.
Abstract
Pretrained models have revolutionized deep learning by enabling significant performance improvements across a wide range of tasks, leveraging large-scale, pre-learned knowledge representations. However, deploying these models in real-world multi-task learning (MTL) scenarios poses substantial challenges, primarily due to high computational costs and inefficiencies in inference. Traditional approaches such as pruning, quantization, and knowledge distillation have been explored to mitigate these issues, but they often fall short in fully addressing the complexities of multi-task environments. This paper introduces \textbf{\underline{S}}elective \textbf{\underline{T}}ask \textbf{\underline{A}}rithmetic \underline{\textbf{(STA)}}, a training-free framework designed to enhance multi-task performance through task-specific parameter fusion. STA addresses three key challenges: (i)…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsParallel Computing and Optimization Techniques · Scheduling and Optimization Algorithms · Distributed and Parallel Computing Systems
MethodsKnowledge Distillation
