Towards the Unification of Generative and Discriminative Visual Foundation Model: A Survey

TL;DR

This survey reviews the development of visual foundation models, highlighting their capabilities in generative and discriminative tasks, and discusses the potential for unifying these paradigms to advance computer vision.

Contribution

It provides a comprehensive overview of VFMs, their key breakthroughs, resources, challenges, and emphasizes the emerging integration of generative and discriminative approaches.

Findings

VFMs exhibit strong zero-shot generalization capabilities.

Recent advances enable VFMs to perform both generative and discriminative tasks.

The field is moving towards unifying generative and discriminative paradigms.

Abstract

The advent of foundation models, which are pre-trained on vast datasets, has ushered in a new era of computer vision, characterized by their robustness and remarkable zero-shot generalization capabilities. Mirroring the transformative impact of foundation models like large language models (LLMs) in natural language processing, visual foundation models (VFMs) have become a catalyst for groundbreaking developments in computer vision. This review paper delineates the pivotal trajectories of VFMs, emphasizing their scalability and proficiency in generative tasks such as text-to-image synthesis, as well as their adeptness in discriminative tasks including image segmentation. While generative and discriminative models have historically charted distinct paths, we undertake a comprehensive examination of the recent strides made by VFMs in both domains, elucidating their origins, seminal breakthroughs, and pivotal methodologies. Additionally, we collate and discuss the extensive resources that facilitate the development of VFMs and address the challenges that pave the way for future research endeavors. A crucial direction for forthcoming innovation is the amalgamation of generative and discriminative paradigms. The nascent application of generative models within discriminative contexts signifies the early stages of this confluence. This survey aspires to be a contemporary compendium for scholars and practitioners alike, charting the course of VFMs and illuminating their multifaceted landscape.