VisionGPT: Vision-Language Understanding Agent Using Generalized Multimodal Framework

TL;DR

VisionGPT is a versatile framework that combines large language models with vision foundation models to enable open-world visual perception and comprehensive vision-language understanding across various applications.

Contribution

It introduces a generalized multimodal framework that automates the integration of multiple foundation models using LLMs as a central coordinator, enhancing flexibility and performance.

Findings

Effective integration of multiple foundation models for vision tasks.

Versatile applications including image understanding, editing, and visual question answering.

Demonstrated potential to advance open-world visual perception.

Abstract

With the emergence of large language models (LLMs) and vision foundation models, how to combine the intelligence and capacity of these open-sourced or API-available models to achieve open-world visual perception remains an open question. In this paper, we introduce VisionGPT to consolidate and automate the integration of state-of-the-art foundation models, thereby facilitating vision-language understanding and the development of vision-oriented AI. VisionGPT builds upon a generalized multimodal framework that distinguishes itself through three key features: (1) utilizing LLMs (e.g., LLaMA-2) as the pivot to break down users' requests into detailed action proposals to call suitable foundation models; (2) integrating multi-source outputs from foundation models automatically and generating comprehensive responses for users; (3) adaptable to a wide range of applications such as text-conditioned image understanding/generation/editing and visual question answering. This paper outlines the architecture and capabilities of VisionGPT, demonstrating its potential to revolutionize the field of computer vision through enhanced efficiency, versatility, and generalization, and performance. Our code and models will be made publicly available. Keywords: VisionGPT, Open-world visual perception, Vision-language understanding, Large language model, and Foundation model