DiTASK: Multi-Task Fine-Tuning with Diffeomorphic Transformations

TL;DR

DiTASK introduces a novel diffeomorphic multi-task fine-tuning method for Vision Transformers that preserves pre-trained representations while enabling efficient task-specific adaptations, achieving state-of-the-art results with fewer parameters.

Contribution

DiTASK proposes a new approach that maintains the geometric structure of pre-trained models during multi-task fine-tuning using neural diffeomorphic transformations.

Findings

Achieves state-of-the-art performance on PASCAL MTL and NYUD datasets.

Uses 75% fewer parameters than existing multi-task learning methods.

Maintains full-rank updates, preserving the geometric structure of features.

Abstract

Pre-trained Vision Transformers now serve as powerful tools for computer vision. Yet, efficiently adapting them for multiple tasks remains a challenge that arises from the need to modify the rich hidden representations encoded by the learned weight matrices, without inducing interference between tasks. Current parameter-efficient methods like LoRA, which apply low-rank updates, force tasks to compete within constrained subspaces, ultimately degrading performance. We introduce DiTASK a novel Diffeomorphic Multi-Task Fine-Tuning approach that maintains pre-trained representations by preserving weight matrix singular vectors, while enabling task-specific adaptations through neural diffeomorphic transformations of the singular values. By following this approach, DiTASK enables both shared and task-specific feature modulations with minimal added parameters. Our theoretical analysis shows that DITASK achieves full-rank updates during optimization, preserving the geometric structure of pre-trained features, and establishing a new paradigm for efficient multi-task learning (MTL). Our experiments on PASCAL MTL and NYUD show that DiTASK achieves state-of-the-art performance across four dense prediction tasks, using 75% fewer parameters than existing methods. Our code is available [here](https://github.com/ipsitmantri/DiTASK).