BRAVE: Broadening the visual encoding of vision-language models

TL;DR

BRAVE enhances vision-language models by integrating multiple visual encodings to improve performance and address limitations like visual hallucination, achieving state-of-the-art results with fewer trainable parameters.

Contribution

The paper introduces BRAVE, a novel method that consolidates features from multiple frozen encoders into a versatile representation for VLMs, improving performance and robustness.

Findings

BRAVE outperforms existing methods on captioning and VQA benchmarks.

Using multiple encoders broadens visual understanding and reduces hallucinations.

BRAVE requires fewer trainable parameters and offers a more compressed representation.

Abstract

Vision-language models (VLMs) are typically composed of a vision encoder, e.g. CLIP, and a language model (LM) that interprets the encoded features to solve downstream tasks. Despite remarkable progress, VLMs are subject to several shortcomings due to the limited capabilities of vision encoders, e.g. "blindness" to certain image features, visual hallucination, etc. To address these issues, we study broadening the visual encoding capabilities of VLMs. We first comprehensively benchmark several vision encoders with different inductive biases for solving VLM tasks. We observe that there is no single encoding configuration that consistently achieves top performance across different tasks, and encoders with different biases can perform surprisingly similarly. Motivated by this, we introduce a method, named BRAVE, that consolidates features from multiple frozen encoders into a more versatile representation that can be directly fed as the input to a frozen LM. BRAVE achieves state-of-the-art performance on a broad range of captioning and VQA benchmarks and significantly reduces the aforementioned issues of VLMs, while requiring a smaller number of trainable parameters than existing methods and having a more compressed representation. Our results highlight the potential of incorporating different visual biases for a more broad and contextualized visual understanding of VLMs.