Investigating Cross-Modal Skill Injection: Scenarios, Methods, and Hyperparameters

TL;DR

This paper systematically analyzes cross-modal skill injection in vision-language models, focusing on scenarios, methods, and hyperparameters, to enhance understanding and effectiveness of this transfer technique.

Contribution

It provides a comprehensive evaluation of cross-modal skill injection, identifying effective scenarios, comparing merging methods, and analyzing hyperparameter impacts.

Findings

Cross-modal skill injection performs well in instruction-following and cross-lingual tasks.

Classic merging methods like TA and DARE outperform alternatives.

Hyperparameter tuning is critical for the success of merging methods.

Abstract

Vision-Language Models (VLMs) have demonstrated remarkable proficiency in general multi-modal understanding; yet they struggle to efficiently acquire continually evolving domain-specific skills. Conventional approaches to enhancing VLM capabilities, such as Supervised Fine-Tuning (SFT), require extensive dataset curation and substantial computational resources. Model merging has emerged as an efficient alternative that enables the transfer of domain-specific expertise from Large Language Models (LLMs) to VLMs without incurring additional training data requirements or significant computational overhead. Unlike conventional merging of homogeneous LLMs, which mainly aggregates existing capabilities, cross-modal skill injection aims to induce emergent cross-modal capabilities by integrating a domain-expert LLM into a VLM. However, existing research lacks a systematic analysis of the applicability and methodology of cross-modal skill injection. In this study, we investigate cross-modal skill injection across three main aspects: scenarios, methods, and hyperparameters. For scenarios, we find that cross-modal skill injection generally performs well in instruction-following and cross-lingual settings, yet struggles with mathematical reasoning. For methods, we find that classic approaches such as TA and DARE consistently achieve superior performance over alternative merging methods. We also provide a systematic and quantitative analysis of the hyperparameter tuning that these classic methods critically depend on.