GiVE: Guiding Visual Encoder to Perceive Overlooked Information

TL;DR

GiVE introduces a novel visual encoder enhancement with specialized modules and loss functions, significantly improving object perception and retrieval in multimodal models, leading to state-of-the-art results.

Contribution

The paper presents GiVE, a new visual encoder framework with attention-guided modules and loss functions, plus a new dataset, to better perceive overlooked objects in multimodal tasks.

Findings

Achieves state-of-the-art performance on relevant benchmarks.

Enhances object retrieval accuracy and comprehensiveness.

Improves visual focus adjustment in multimodal models.

Abstract

Multimodal Large Language Models have advanced AI in applications like text-to-video generation and visual question answering. These models rely on visual encoders to convert non-text data into vectors, but current encoders either lack semantic alignment or overlook non-salient objects. We propose the Guiding Visual Encoder to Perceive Overlooked Information (GiVE) approach. GiVE enhances visual representation with an Attention-Guided Adapter (AG-Adapter) module and an Object-focused Visual Semantic Learning module. These incorporate three novel loss terms: Object-focused Image-Text Contrast (OITC) loss, Object-focused Image-Image Contrast (OIIC) loss, and Object-focused Image Discrimination (OID) loss, improving object consideration, retrieval accuracy, and comprehensiveness. Our contributions include dynamic visual focus adjustment, novel loss functions to enhance object retrieval, and the Multi-Object Instruction (MOInst) dataset. Experiments show our approach achieves state-of-the-art performance.