Multi-Modal Adapter for Vision-Language Models

TL;DR

This paper introduces Multi-Modal Adapter, a novel lightweight method that enhances vision-language models like CLIP by integrating visual and textual features through multi-head attention, improving generalization to unseen classes.

Contribution

The paper proposes a Multi-Modal Adapter with a trainable attention layer for better integration of visual and textual features in CLIP, advancing adaptation techniques.

Findings

Improved performance on unseen classes compared to existing methods

Enhanced generalizability of CLIP through multi-modal adaptation

Validated approach with ablation studies and interpretability analysis

Abstract

Large pre-trained vision-language models, such as CLIP, have demonstrated state-of-the-art performance across a wide range of image classification tasks, without requiring retraining. Few-shot CLIP is competitive with existing specialized architectures that were trained on the downstream tasks. Recent research demonstrates that the performance of CLIP can be further improved using lightweight adaptation approaches. However, previous methods adapt different modalities of the CLIP model individually, ignoring the interactions and relationships between visual and textual representations. In this work, we propose Multi-Modal Adapter, an approach for Multi-Modal adaptation of CLIP. Specifically, we add a trainable Multi-Head Attention layer that combines text and image features to produce an additive adaptation of both. Multi-Modal Adapter demonstrates improved generalizability, based on its performance on unseen classes compared to existing adaptation methods. We perform additional ablations and investigations to validate and interpret the proposed approach.