TADFormer : Task-Adaptive Dynamic Transformer for Efficient Multi-Task Learning

TL;DR

TADFormer introduces a task-adaptive dynamic transformer that enhances multi-task learning efficiency by dynamically capturing task-specific features, achieving higher accuracy with fewer trainable parameters.

Contribution

The paper presents a novel PEFT framework with dynamic task filtering and prompting, improving task-specific feature adaptation in multi-task learning.

Findings

Achieves higher accuracy on PASCAL-Context benchmark

Reduces trainable parameters by up to 8.4 times

Outperforms recent PEFT methods in efficiency and accuracy

Abstract

Transfer learning paradigm has driven substantial advancements in various vision tasks. However, as state-of-the-art models continue to grow, classical full fine-tuning often becomes computationally impractical, particularly in multi-task learning (MTL) setup where training complexity increases proportional to the number of tasks. Consequently, recent studies have explored Parameter-Efficient Fine-Tuning (PEFT) for MTL architectures. Despite some progress, these approaches still exhibit limitations in capturing fine-grained, task-specific features that are crucial to MTL. In this paper, we introduce Task-Adaptive Dynamic transFormer, termed TADFormer, a novel PEFT framework that performs task-aware feature adaptation in the fine-grained manner by dynamically considering task-specific input contexts. TADFormer proposes the parameter-efficient prompting for task adaptation and the Dynamic Task Filter (DTF) to capture task information conditioned on input contexts. Experiments on the PASCAL-Context benchmark demonstrate that the proposed method achieves higher accuracy in dense scene understanding tasks, while reducing the number of trainable parameters by up to 8.4 times when compared to full fine-tuning of MTL models. TADFormer also demonstrates superior parameter efficiency and accuracy compared to recent PEFT methods.