LMM-driven Semantic Image-Text Coding for Ultra Low-bitrate Learned Image Compression

TL;DR

This paper introduces a novel semantic image-text coding method driven by large multi-modal models, significantly improving low-bitrate image compression quality by integrating caption generation and compression within a unified framework.

Contribution

It presents a new approach that uses LMMs to generate and compress captions simultaneously, along with a semantic-perceptual fine-tuning method that enhances compression performance.

Findings

Achieved 41.58% improvement in LPIPS BD-rate over existing methods

Demonstrated effective integration of caption generation and image compression

Provided open-source implementation and pre-trained models

Abstract

Supported by powerful generative models, low-bitrate learned image compression (LIC) models utilizing perceptual metrics have become feasible. Some of the most advanced models achieve high compression rates and superior perceptual quality by using image captions as sub-information. This paper demonstrates that using a large multi-modal model (LMM), it is possible to generate captions and compress them within a single model. We also propose a novel semantic-perceptual-oriented fine-tuning method applicable to any LIC network, resulting in a 41.58\% improvement in LPIPS BD-rate compared to existing methods. Our implementation and pre-trained weights are available at https://github.com/tokkiwa/ImageTextCoding.