Advancing Visual Large Language Model for Multi-granular Versatile Perception

TL;DR

This paper introduces MVP-LM, a versatile visual perception framework using a large language model that unifies multiple tasks like detection, segmentation, and grounding within a single architecture, enhancing adaptability and performance.

Contribution

The paper presents MVP-LM, a novel multi-granular, versatile perception framework that integrates various vision tasks with a unified decoder and dataset unification strategy.

Findings

Effective across diverse benchmarks

Unifies multiple perception tasks in a single model

Demonstrates superior performance over existing methods

Abstract

Perception is a fundamental task in the field of computer vision, encompassing a diverse set of subtasks that can be systematically categorized into four distinct groups based on two dimensions: prediction type and instruction type. Notably, existing researches often focus solely on a limited subset of these potential combinations, which constrains their applicability and versatility across various contexts. In response to this challenge, we present MVP-LM, a Multi-granular and Versatile Perception framework incorporating Visual Large Language Model. Our framework is designed to integrate both word-based and sentence-based perception tasks alongside box and mask predictions within a single architecture. MVP-LM features an innovative multi-granularity decoder in conjunction with a CoT-inspired dataset unification strategy, enabling seamless supervised fine-tuning across a wide spectrum of tasks, including but not limited to panoptic segmentation, detection, grounding, and referring expression segmentation. Furthermore, we introduce a query enhancement strategy aimed at harnessing the decoding and generative capabilities inherent in VLLMs. Extensive experiments conducted across a range of benchmarks in both word-based and sentence-based perception tasks substantiate the efficacy of our framework. The code will be available at https://github.com/xiangwentao666/MVP-LM.