Task Alignment: A simple and effective proxy for model merging in computer vision

TL;DR

This paper introduces the task alignment proxy, a method to efficiently evaluate and improve model merging for diverse vision tasks, extending beyond traditional CLIP classification scenarios.

Contribution

The authors propose the task alignment proxy to accelerate hyperparameter tuning and enable practical model merging across heterogeneous multi-task vision models.

Findings

Task alignment proxy significantly speeds up hyperparameter selection.

Model merging effectiveness extends to multi-task vision models beyond CLIP.

The approach maintains performance while reducing computational costs.

Abstract

Efficiently merging several models fine-tuned for different tasks, but stemming from the same pretrained base model, is of great practical interest. Despite extensive prior work, most evaluations of model merging in computer vision are restricted to image classification using CLIP, where different classification datasets define different tasks. In this work, our goal is to make model merging more practical and show its relevance on challenging scenarios beyond this specific setting. In most vision scenarios, different tasks rely on trainable and usually heterogeneous decoders. Differently from previous studies with frozen decoders, where merged models can be evaluated right away, the non-trivial cost of decoder training renders hyperparameter selection based on downstream performance impractical. To address this, we introduce the task alignment proxy, and show how it can be used to speed up hyperparameter selection by orders of magnitude while retaining performance. Equipped with the task alignment proxy, we extend the applicability of model merging to multi-task vision models beyond CLIP-based classification.