Panther: Illuminate the Sight of Multimodal LLMs with Instruction-Guided Visual Prompts

TL;DR

Panther is a multimodal large language model that enhances visual perception by integrating user instructions early in the vision encoder, reducing redundant information, and accurately locating small objects, especially on vision-centric benchmarks.

Contribution

Introduces Panther, a novel MLLM with instruction-guided visual prompts, featuring modules that improve visual focus and reduce training costs without restricting decoder architecture.

Findings

Effective on vision-centric benchmarks

Improves accuracy in locating small objects

Reduces training costs significantly

Abstract

Multimodal large language models (MLLMs) are closing the gap to human visual perception capability rapidly, while, still lag behind on attending to subtle images details or locating small objects precisely, etc. Common schemes to tackle these issues include deploying multiple vision encoders or operating on original high-resolution images. Few studies have concentrated on taking the textual instruction into improving visual representation, resulting in losing focus in some vision-centric tasks, a phenomenon we herein termed as Amblyopia. In this work, we introduce Panther, a MLLM that closely adheres to user instruction and locates targets of interests precisely, with the finesse of a black panther. Specifically, Panther comprises three integral components: Panther-VE, Panther-Bridge, and Panther-Decoder. Panther-VE integrates user instruction information at the early stages of the vision encoder, thereby extracting the most relevant and useful visual representations. The Panther-Bridge module, equipped with powerful filtering capabilities, significantly reduces redundant visual information, leading to a substantial savings in training costs. The Panther-Decoder is versatile and can be employed with any decoder-only architecture of LLMs without discrimination. Experimental results, particularly on vision-centric benchmarks, have demonstrated the effectiveness of Panther.