High Efficiency Image Compression for Large Visual-Language Models

TL;DR

This paper introduces a variable bitrate image compression framework optimized for large visual-language models, enhancing rate-accuracy performance and generalization across multi-modal tasks.

Contribution

It proposes a novel optimization strategy and a joint training framework for image compression tailored specifically for LVLMs, improving efficiency and versatility.

Findings

Outperforms Versatile Video Coding in rate-accuracy performance

Demonstrates robustness across various multi-modal tasks

Enhances generalization capability for different data types

Abstract

In recent years, large visual language models (LVLMs) have shown impressive performance and promising generalization capability in multi-modal tasks, thus replacing humans as receivers of visual information in various application scenarios. In this paper, we pioneer to propose a variable bitrate image compression framework consisting of a pre-editing module and an end-to-end codec to achieve promising rate-accuracy performance for different LVLMs. In particular, instead of optimizing an adaptive pre-editing network towards a particular task or several representative tasks, we propose a new optimization strategy tailored for LVLMs, which is designed based on the representation and discrimination capability with token-level distortion and rank. The pre-editing module and the variable bitrate end-to-end image codec are jointly trained by the losses based on semantic tokens of the large model, which introduce enhanced generalization capability for various data and tasks. {Experimental results demonstrate that the proposed framework could efficiently achieve much better rate-accuracy performance compared to the state-of-the-art coding standard, Versatile Video Coding.} Meanwhile, experiments with multi-modal tasks have revealed the robustness and generalization capability of the proposed framework.