Gate-and-Merge: Zero-shot Compositional Personalization of Vision Language Models

TL;DR

This paper presents Gate-and-Merge, a zero-shot framework for compositional personalization of vision-language models that combines independently learned concepts during inference without retraining.

Contribution

It introduces a novel zero-shot method that enables compositional personalization by merging concept-specific updates with gating, without requiring co-occurrence training.

Findings

Achieves consistent performance gains in personalization tasks

Effectively merges concepts without interference

Maintains concept identity during composition

Abstract

This paper tackles compositional personalization of vision-language models (VLMs). In this problem, multiple user-defined concepts must be recognized or described jointly at test time. We introduce Gate-and-Merge, a zero-shot framework that enables compositional personalization without the need for co-occurrence training. During personalization, each concept is learned independently as a lightweight LoRA adapter, paired with a concept token. The base model remains unchanged and concepts are kept disentangled. At inference, we enable composition by merging concept-specific LoRA updates directly in weight space. To suppress irrelevant activations and prevent interference, a gating mechanism is employed to estimate textual and visual cues and select only the modules that contribute to the prediction. We further stabilize composition by combining only the most meaningful and mutually consistent updates, helping preserve each concept's identity. Our quantitative and qualitative analyses show consistent gains in performance across multiple personalization tasks in both single-concept and compositional settings.