Feast Your Eyes: Mixture-of-Resolution Adaptation for Multimodal Large Language Models

TL;DR

This paper introduces Mixture-of-Resolution Adaptation (MRA), a novel method for multimodal large language models that combines low- and high-resolution visual features to improve granular visual recognition efficiently.

Contribution

It proposes MRA, a new approach using dual visual pathways with resolution adapters, enhancing performance and efficiency of MLLMs like LLaVA-HR.

Findings

LLaVA-HR outperforms existing MLLMs on 8 vision-language tasks.

MRA reduces training time to 20 hours and triples inference speed.

Combining resolutions improves granular visual recognition in MLLMs.

Abstract

Despite remarkable progress, existing multimodal large language models (MLLMs) are still inferior in granular visual recognition. Contrary to previous works, we study this problem from the perspective of image resolution, and reveal that a combination of low- and high-resolution visual features can effectively mitigate this shortcoming. Based on this observation, we propose a novel and efficient method for MLLMs, termed Mixture-of-Resolution Adaptation (MRA). In particular, MRA adopts two visual pathways for images with different resolutions, where high-resolution visual information is embedded into the low-resolution pathway via the novel mixture-of-resolution adapters (MR-Adapters). This design also greatly reduces the input sequence length of MLLMs. To validate MRA, we apply it to a recent MLLM called LLaVA, and term the new model LLaVA-HR. We conduct extensive experiments on 11 vision-language (VL) tasks, which show that LLaVA-HR outperforms existing MLLMs on 8 VL tasks, e.g., +9.4% on TextVQA. More importantly, both training and inference of LLaVA-HR remain efficient with MRA, e.g., 20 training hours and 3$\times$ inference speed than LLaVA-1.5. Source codes are released at: https://github.com/luogen1996/LLaVA-HR.