Frequency Switching Mechanism for Parameter-E!cient Multi-Task Learning

TL;DR

The paper introduces Free Sinewich, a novel parameter-efficient multi-task learning framework that uses frequency switching and sinusoidal modulation to enable scalable, task-specific weights with minimal additional parameters.

Contribution

It proposes a new frequency-based modulation method for multi-task learning that improves efficiency and performance over existing single-task fine-tuning approaches.

Findings

Achieves state-of-the-art performance-efficiency trade-offs on dense prediction benchmarks.

Up to +5.39% improvement over single-task fine-tuning with only 6.53M trainable parameters.

Enhances the rank of low-rank adapters and decorrelates task weights through sine modulation.

Abstract

Multi-task learning (MTL) aims to enable a single model to solve multiple tasks efficiently; however, current parameter-efficient fine-tuning (PEFT) methods remain largely limited to single-task adaptation. We introduce \textbf{Free Sinewich}, a parameter-efficient multi-task learning framework that enables near-zero-cost weight modulation via frequency switching (\textbf{Free}). Specifically, a \textbf{Sine-AWB (Sinewich)} layer combines low-rank factors and convolutional priors into a single kernel, which is then modulated elementwise by a sinusoidal transformation to produce task-specialized weights. A lightweight Clock Net is introduced to produce bounded frequencies that stabilize this modulation during training. Theoretically, sine modulation enhances the rank of low-rank adapters, while frequency separation decorrelates the weights of different tasks. On dense prediction benchmarks, Free Sinewich achieves state-of-the-art performance-efficiency trade-offs (e.g., up to +5.39\% improvement over single-task fine-tuning with only 6.53M trainable parameters), offering a compact and scalable paradigm based on frequency-based parameter sharing. Project page: \href{https://casperliuliuliu.github.io/projects/Free-Sinewich/}{https://casperliuliuliu.github.io/projects/Free-Sinewich}.