Vision-Language Models for Vision Tasks: A Survey

TL;DR

This survey reviews the development, architectures, datasets, methods, and challenges of vision-language models (VLMs) that enable zero-shot visual recognition, highlighting their potential to transform traditional paradigms.

Contribution

It systematically categorizes existing VLM pre-training, transfer learning, and knowledge distillation methods, providing comprehensive analysis and future research directions.

Findings

VLMs enable zero-shot recognition across various tasks.

Pre-training datasets and architectures significantly impact VLM performance.

Benchmarking reveals strengths and limitations of current VLM approaches.

Abstract

Most visual recognition studies rely heavily on crowd-labelled data in deep neural networks (DNNs) training, and they usually train a DNN for each single visual recognition task, leading to a laborious and time-consuming visual recognition paradigm. To address the two challenges, Vision-Language Models (VLMs) have been intensively investigated recently, which learns rich vision-language correlation from web-scale image-text pairs that are almost infinitely available on the Internet and enables zero-shot predictions on various visual recognition tasks with a single VLM. This paper provides a systematic review of visual language models for various visual recognition tasks, including: (1) the background that introduces the development of visual recognition paradigms; (2) the foundations of VLM that summarize the widely-adopted network architectures, pre-training objectives, and downstream tasks; (3) the widely-adopted datasets in VLM pre-training and evaluations; (4) the review and categorization of existing VLM pre-training methods, VLM transfer learning methods, and VLM knowledge distillation methods; (5) the benchmarking, analysis and discussion of the reviewed methods; (6) several research challenges and potential research directions that could be pursued in the future VLM studies for visual recognition. A project associated with this survey has been created at https://github.com/jingyi0000/VLM_survey.