Generalizing vision-language models to novel domains: A comprehensive survey

TL;DR

This survey reviews recent advances in vision-language models, focusing on their generalization to new domains, categorizing methods, benchmarking results, and relation to multimodal large language models, to guide future research.

Contribution

It provides a comprehensive categorization and analysis of VLM generalization methods, benchmarks, and their relation to multimodal large language models, offering a clear research landscape.

Findings

Categorizes VLM generalization methods into prompt-, parameter-, and feature-based.

Summarizes key benchmarks and compares method performances.

Discusses the relationship between VLMs and multimodal large language models.

Abstract

Recently, vision-language pretraining has emerged as a transformative technique that integrates the strengths of both visual and textual modalities, resulting in powerful vision-language models (VLMs). Leveraging web-scale pretraining data, these models exhibit strong zero-shot capabilities. However, their performance often deteriorates when confronted with domain-specific or specialized generalization tasks. To address this, a growing body of research focuses on transferring or generalizing the rich knowledge embedded in VLMs to various downstream applications. This survey aims to comprehensively summarize the generalization settings, methodologies, benchmarking and results in VLM literatures. Delving into the typical VLM structures, current literatures are categorized into prompt-based, parameter-based and feature-based methods according to the transferred modules. The differences and characteristics in each category are furthered summarized and discussed by revisiting the typical transfer learning (TL) settings, providing novel interpretations for TL in the era of VLMs. Popular benchmarks for VLM generalization are further introduced with thorough performance comparisons among the reviewed methods. Following the advances in large-scale generalizable pretraining, this survey also discusses the relations and differences between VLMs and up-to-date multimodal large language models (MLLM), e.g., DeepSeek-VL. By systematically reviewing the surging literatures in vision-language research from a novel and practical generalization prospective, this survey contributes to a clear landscape of current and future multimodal researches.