GRiT: A Generative Region-to-text Transformer for Object Understanding

TL;DR

GRiT is a transformer-based model that jointly performs object detection and dense captioning by generating descriptive text for object regions, enabling richer object understanding.

Contribution

It introduces a unified generative framework that models object understanding as region-to-text pairs, capable of producing both class labels and descriptive sentences.

Findings

Achieves 60.4 AP on COCO detection

Attains 15.5 mAP on Visual Genome captioning

Unifies detection and captioning in a single model

Abstract

This paper presents a Generative RegIon-to-Text transformer, GRiT, for object understanding. The spirit of GRiT is to formulate object understanding as <region, text> pairs, where region locates objects and text describes objects. For example, the text in object detection denotes class names while that in dense captioning refers to descriptive sentences. Specifically, GRiT consists of a visual encoder to extract image features, a foreground object extractor to localize objects, and a text decoder to generate open-set object descriptions. With the same model architecture, GRiT can understand objects via not only simple nouns, but also rich descriptive sentences including object attributes or actions. Experimentally, we apply GRiT to object detection and dense captioning tasks. GRiT achieves 60.4 AP on COCO 2017 test-dev for object detection and 15.5 mAP on Visual Genome for dense captioning. Code is available at https://github.com/JialianW/GRiT