CMC-Bench: Towards a New Paradigm of Visual Signal Compression
Chunyi Li, Xiele Wu, Haoning Wu, Donghui Feng, Zicheng Zhang, Guo Lu,, Xiongkuo Min, Xiaohong Liu, Guangtao Zhai, Weisi Lin
TL;DR
This paper introduces CMC-Bench, a benchmark for evaluating the cooperative performance of image-to-text and text-to-image models in ultra-low bitrate image compression, demonstrating potential advantages over traditional codecs.
Contribution
It establishes a comprehensive benchmark for cross-modality compression models, providing a large dataset and subjective evaluations to guide future improvements.
Findings
Some I2T and T2I model combinations outperform traditional codecs at ultra-low bitrates.
The benchmark includes 160,000 subjective preference scores from human experts.
The study identifies areas where LMMs can be optimized for better compression performance.
Abstract
Ultra-low bitrate image compression is a challenging and demanding topic. With the development of Large Multimodal Models (LMMs), a Cross Modality Compression (CMC) paradigm of Image-Text-Image has emerged. Compared with traditional codecs, this semantic-level compression can reduce image data size to 0.1\% or even lower, which has strong potential applications. However, CMC has certain defects in consistency with the original image and perceptual quality. To address this problem, we introduce CMC-Bench, a benchmark of the cooperative performance of Image-to-Text (I2T) and Text-to-Image (T2I) models for image compression. This benchmark covers 18,000 and 40,000 images respectively to verify 6 mainstream I2T and 12 T2I models, including 160,000 subjective preference scores annotated by human experts. At ultra-low bitrates, this paper proves that the combination of some I2T and T2I models…
Peer Reviews
Decision·Submitted to ICLR 2025
1. The idea of conducting a benchmark to evaluate CMC-based codecs is valuable. 2. This paper provides a comprehensive set of experiments studying various combinations of I2T and T2I models.
1. This paper does not propose a new model but instead (1) creates a mixed dataset by collecting images from several existing datasets and (2) combines existing I2T and T2I models to evaluate their coding performance. The technical novelty of this paper is therefore questionable. 2. The reported -FR and -NR values are the weighted average of multiple metrics, but it is unclear how the proposed weighting ensures a reasonable assessment. 3. In Fig. 6, the gap between the upper and lower bounds of
I appreciate the effort that the authors put in annotating subjective scores and building such a relatively large dataset for image compression.
1. Throughout the paper, I did not find any non-trivial insight from the proposed benchmark for the whole community. The main conclusion of the proposed benchmark assessment is that: in the task of ultra-low rate compression, codecs based on generative models (e.g., T2I models) could perform better than traditional codecs in many aspects. However, actually it is almost a common sense in the compression community, I would suggest the authors to think deeper so that it can get some valuable conclu
This paper introduces a new benchmark platform for Cross Modality Compression (CMC) called CMC-Bench, offering a fresh approach and guiding direction for future ultra-low bitrate image compression. The paper includes 58,000 images and 160,000 human annotations, offering a large-scale and comprehensive evaluation. Furthermore, authors presents rigorous comparative experiments with existing traditional codecs, demonstrating the advantages of CMC, which is highly persuasive. Although it does not pr
Some details require further discussion, such as the balance between consistency and perceptual quality. Additionally, certain expressions and the structure of the paper would benefit from further refinement and polishing.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Data Compression Techniques · Advanced Vision and Imaging · Image Retrieval and Classification Techniques