Vision Generalist Model: A Survey

TL;DR

This survey comprehensively reviews vision generalist models, discussing their design, capabilities, challenges, and future directions, highlighting their emerging role in computer vision tasks inspired by successes in NLP.

Contribution

It provides a detailed overview of vision generalist models, including their frameworks, techniques, applications, and challenges, serving as a foundational resource for future research in the field.

Findings

Vision generalist models can handle diverse vision tasks.

Current frameworks face challenges in input-output diversity.

Future research directions include improving model versatility.

Abstract

Recently, we have witnessed the great success of the generalist model in natural language processing. The generalist model is a general framework trained with massive data and is able to process various downstream tasks simultaneously. Encouraged by their impressive performance, an increasing number of researchers are venturing into the realm of applying these models to computer vision tasks. However, the inputs and outputs of vision tasks are more diverse, and it is difficult to summarize them as a unified representation. In this paper, we provide a comprehensive overview of the vision generalist models, delving into their characteristics and capabilities within the field. First, we review the background, including the datasets, tasks, and benchmarks. Then, we dig into the design of frameworks that have been proposed in existing research, while also introducing the techniques employed to enhance their performance. To better help the researchers comprehend the area, we take a brief excursion into related domains, shedding light on their interconnections and potential synergies. To conclude, we provide some real-world application scenarios, undertake a thorough examination of the persistent challenges, and offer insights into possible directions for future research endeavors.