Efficient Multi-modal Large Language Models via Visual Token Grouping

TL;DR

This paper introduces VisToG, a grouping mechanism for multi-modal large language models that reduces inference time by over 27% while maintaining high performance, by efficiently compressing visual tokens using pre-trained vision encoders.

Contribution

VisToG is a novel visual token grouping method that leverages pre-trained vision encoders to reduce computational costs without segmentation masks.

Findings

Maintains 98.1% of original performance.

Reduces inference time by over 27%.

Effectively compresses visual tokens in MLLMs.

Abstract

The development of Multi-modal Large Language Models (MLLMs) enhances Large Language Models (LLMs) with the ability to perceive data formats beyond text, significantly advancing a range of downstream applications, such as visual question answering and image captioning. However, the substantial computational costs associated with processing high-resolution images and videos pose a barrier to their broader adoption. To address this challenge, compressing vision tokens in MLLMs has emerged as a promising approach to reduce inference costs. While existing methods conduct token reduction in the feature alignment phase. In this paper, we introduce VisToG, a novel grouping mechanism that leverages the capabilities of pre-trained vision encoders to group similar image segments without the need for segmentation masks. Specifically, we concatenate semantic tokens to represent image semantic segments after the linear projection layer before feeding into the vision encoder. Besides, with the isolated attention we adopt, VisToG can identify and eliminate redundant visual tokens utilizing the prior knowledge in the pre-trained vision encoder, which effectively reduces computational demands. Extensive experiments demonstrate the effectiveness of VisToG, maintaining 98.1% of the original performance while achieving a reduction of over 27\% inference time.