ES-Merging: Biological MLLM Merging via Embedding Space Signals

TL;DR

This paper introduces ES-Merging, a novel embedding space signal-based method for merging specialized biological multimodal language models, enabling effective cross-modal scientific problem solving and outperforming existing approaches.

Contribution

It proposes a representation-aware merging framework that estimates layer-wise and element-wise coefficients from embedding responses, improving model merging fidelity.

Findings

Outperforms existing merging methods on interactive effect prediction benchmarks.

Surpasses task-specific fine-tuned models in performance.

Demonstrates the effectiveness of embedding space signals for model merging.

Abstract

Biological multimodal large language models (MLLMs) have emerged as powerful foundation models for scientific discovery. However, existing models are specialized to a single modality, limiting their ability to solve inherently cross-modal scientific problems. While model merging is an efficient method to combine the different modalities into a unified MLLM, existing methods rely on input-agnostic parameter space heuristics that fail to faithfully capture modality specialization. To overcome this limitation, we propose a representation-aware merging framework that estimates merging coefficients from embedding space signals. We first design a probe input that consists of different modality tokens and forward it through each specialized MLLM to obtain layer-wise embedding responses that reflect modality-specific representation changes. We then estimate complementary merging coefficients at two granularities from the embedding space: layer-wise coefficients from coarse-grained signals and element-wise coefficients from fine-grained signals, which are jointly combined for robust coefficient estimation. Experiments on interactive effect prediction benchmarks show that our method outperforms existing merging methods and even surpasses task-specific fine-tuned models, establishing that embedding space signals provide a principled and effective foundation for cross-modal MLLM merging.