HIRE: Lightweight High-Resolution Image Feature Enrichment for Multimodal LLMs

TL;DR

This paper introduces HIRE, a lightweight method for enriching high-resolution image features in multimodal LLMs, achieving comparable performance with significantly reduced computational costs.

Contribution

The paper proposes a shallow feature enricher that enhances high-resolution features efficiently, reducing FLOPs by up to 1.5x while maintaining competitive accuracy.

Findings

Achieves high performance on visual understanding benchmarks.

Reduces training and inference time significantly.

Maintains competitive results with fewer computational resources.

Abstract

The integration of high-resolution image features in modern multimodal large language models has demonstrated significant improvements in fine-grained visual understanding tasks, achieving high performance across multiple benchmarks. Since these features are obtained from large image encoders like ViT, they come with a significant increase in computational costs due to multiple calls to these encoders. In this work, we first develop an intuition for feature upsampling as a natural extension of high-resolution feature generation. Through extensive experiments and ablations, we demonstrate how a shallow feature enricher can achieve competitive results with tremendous reductions in training and inference time as well as computational cost, with upto 1.5x saving in FLOPs.