DisTaC: Conditioning Task Vectors via Distillation for Robust Model Merging
Kotaro Yoshida, Yuji Naraki, Takafumi Horie, Ryotaro Shimizu, Hiroki Naganuma
TL;DR
This paper introduces DisTaC, a distillation-based method to pre-condition task vectors, improving the robustness and performance of model merging techniques in multi-task learning, especially under challenging source-model conditions.
Contribution
DisTaC is a novel approach that adjusts task vectors via distillation to enhance model merging robustness against disparities and low confidence issues.
Findings
DisTaC improves merging success on challenging models.
Pre-conditioning with DisTaC yields significant performance gains.
DisTaC addresses vulnerabilities related to task vector norms and model confidence.
Abstract
Model merging has emerged as an efficient and flexible paradigm for multi-task learning, with numerous methods being proposed in recent years. However, these state-of-the-art techniques are typically evaluated on benchmark suites that are highly favorable to model merging, and their robustness in more realistic settings remains largely unexplored. In this work, we first investigate the vulnerabilities of model-merging methods and pinpoint the source-model characteristics that critically underlie them. Specifically, we identify two factors that are particularly harmful to the merging process: (1) disparities in task vector norms, and (2) the low confidence of the source models. To address this issue, we propose DisTaC (Distillation for Task vector Conditioning), a novel method that pre-conditions these problematic task vectors before the merge. DisTaC leverages knowledge distillation to…
Peer Reviews
Decision·ICLR 2026 Poster
1. The paper addresses a novel and highly relevant problem. While most research focuses on improving performance on idealized benchmarks, this work astutely investigates the robustness of merging methods in more practical, pessimistic settings, which is a critical step for real-world applicability. 2. The problem formulation is exceptionally clear and well-motivated. The failure modes are demonstrated with convincing empirical evidence (Figure 1), making the motivation for the proposed solution
1. The empirical evaluation is limited in scope. All experiments are conducted on CLIP/ViT models for vision classification tasks. Given the current prominence of model merging in the context of Large Language Models (LLMs), the absence of any experiments on language tasks is a significant limitation and leaves the generalizability of the findings in question. 2. The investigation of failure modes, while insightful, feels somewhat narrow. The paper focuses on norm disparity and low confidence b
1. The paper provides a sharp empirical and theoretical analysis of robustness shortcomings in existing model merging approaches, uncovering how norm disparity and low confidence can produce significant accuracy degradation after merging. 2. DisTaC is a clear, well-motivated conditioning method, leveraging knowledge distillation with unlabeled data for practical pre-conditioning. By integrating both norm scaling and confidence sharpening, it directly addresses the two key failure modes.
1. Since the pre-conditioning step is applied independently to each source model, the total computational cost scales linearly with the number of models. For large-scale ensembles, this cumulative overhead can become prohibitive, undermining the method's primary efficiency advantage over full-scale retraining. The distillation cost also scales with model parameter count, becoming computationally prohibitive for large-scale foundation models (e.g., LLMs). 2. The study’s empirical validation is c
The experiments are extensive, including ablation studies, visual analyses (e.g., layer-wise norm shifts in Figs. 5–6), and additional tests with Mixup and focal loss to demonstrate generality. The authors go beyond idealized benchmarks and pinpoint practical causes of model-merging failures, which is interesting and meaningful.
The experiments are restricted to vision-only tasks using CLIP ViT backbones. No tests are conducted on NLP, speech, or multimodal architectures. While the paper argues that unlabeled data are “readily available,” this assumption may not hold in all domains, such as the security issue, or we can not find a comprehensive set for LLMs. While DisTaC generally improves results, the paper lacks a sensitivity analysis of unlabeled data quality that would clarify when it might fail or overfit. The a
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsDomain Adaptation and Few-Shot Learning · Adversarial Robustness in Machine Learning · Topic Modeling