Griffon: Spelling out All Object Locations at Any Granularity with Large Language Models

TL;DR

Griffon leverages large vision-language models to accurately locate objects at various granularities without specialized modules, advancing fine-grained object perception and surpassing previous models on multiple benchmarks.

Contribution

Introducing Griffon, a novel LVLM-based approach that unifies data formats and is trained end-to-end, enabling precise object localization without additional detection modules or expert models.

Findings

Achieves state-of-the-art on RefCOCO and Flickr30K Entities

Approaches Faster RCNN performance on MSCOCO detection

Demonstrates LVLMs' basic object perception capabilities

Abstract

Replicating the innate human ability to detect all objects based on free-form texts at any granularity remains a formidable challenge for Large Vision Language Models (LVLMs). Current LVLMs are predominantly constrained to locate a single, pre-existing object. This limitation leads to a compromise in model design, necessitating the introduction of visual expert models or customized head structures. Beyond these constraints, our research uncovers LVLMs' capability for basic object perception, allowing them to accurately identify and locate objects of interest. Building on this insight, we introduce a novel Language-prompted Localization Dataset to fully unleash the capabilities of LVLMs in fine-grained object perception and precise location awareness. More importantly, we present Griffon, a purely LVLM-based baseline, which does not introduce any special tokens, expert models, or additional detection modules. It simply maintains a consistent structure with popular LVLMs by unifying data formats across various localization-related scenarios and is trained end-to-end through a well-designed pipeline. Comprehensive experiments demonstrate that Griffon not only achieves state-of-the-art performance on the fine-grained RefCOCO series and Flickr30K Entities but also approaches the capabilities of the expert model Faster RCNN on the detection benchmark MSCOCO. Data, codes, and models are released at https://github.com/jefferyZhan/Griffon.