FlexCap: Describe Anything in Images in Controllable Detail

TL;DR

FlexCap is a controllable vision-language model that generates region-specific descriptions of images with adjustable detail levels, improving dense captioning and zero-shot visual question answering performance.

Contribution

We introduce FlexCap, a novel model capable of producing length-conditioned region descriptions, enabling controllable detail and enhancing various vision-language tasks.

Findings

Achieves strong dense captioning performance on Visual Genome.

Enables zero-shot VQA with state-of-the-art results.

Supports diverse applications like image labeling and visual dialog.

Abstract

We introduce FlexCap, a vision-language model that generates region-specific descriptions of varying lengths. FlexCap is trained to produce length-conditioned captions for input boxes, enabling control over information density, with descriptions ranging from concise object labels to detailed captions. To achieve this, we create large-scale training datasets of image region descriptions with varying lengths from captioned web images. We demonstrate FlexCap's effectiveness in several applications: first, it achieves strong performance in dense captioning tasks on the Visual Genome dataset. Second, we show how FlexCap's localized descriptions can serve as input to a large language model to create a visual question answering (VQA) system, achieving state-of-the-art zero-shot performance on multiple VQA benchmarks. Our experiments illustrate FlexCap's utility for tasks including image labeling, object attribute recognition, and visual dialog. Project webpage: https://flex-cap.github.io .