Multi-Modality Deep Network for Extreme Learned Image Compression

TL;DR

This paper introduces a multimodal deep learning approach for image compression guided by text, significantly improving visual quality at extremely low bitrates by leveraging semantic information.

Contribution

It proposes a novel text-guided multimodal compression framework with specialized modules and loss functions, outperforming existing methods at lower bitrates.

Findings

Achieves visually pleasing results at extremely low bitrates

Outperforms state-of-the-art methods at 2x to 4x lower bitrates

User studies confirm superior visual quality

Abstract

Image-based single-modality compression learning approaches have demonstrated exceptionally powerful encoding and decoding capabilities in the past few years , but suffer from blur and severe semantics loss at extremely low bitrates. To address this issue, we propose a multimodal machine learning method for text-guided image compression, in which the semantic information of text is used as prior information to guide image compression for better compression performance. We fully study the role of text description in different components of the codec, and demonstrate its effectiveness. In addition, we adopt the image-text attention module and image-request complement module to better fuse image and text features, and propose an improved multimodal semantic-consistent loss to produce semantically complete reconstructions. Extensive experiments, including a user study, prove that our method can obtain visually pleasing results at extremely low bitrates, and achieves a comparable or even better performance than state-of-the-art methods, even though these methods are at 2x to 4x bitrates of ours.