Cross Modal Compression: Towards Human-comprehensible Semantic Compression

TL;DR

This paper introduces cross modal compression (CMC), a novel semantic compression framework that transforms visual data into human-understandable formats, achieving high compression ratios while preserving semantic content.

Contribution

The paper formulates CMC as a rate-distortion problem, compares it with traditional and feature compression, and demonstrates its effectiveness with qualitative and quantitative results.

Findings

CMC achieves higher compression ratios than JPEG.

CMC preserves semantic information effectively.

Qualitative and quantitative evaluations validate CMC's performance.

Abstract

Traditional image/video compression aims to reduce the transmission/storage cost with signal fidelity as high as possible. However, with the increasing demand for machine analysis and semantic monitoring in recent years, semantic fidelity rather than signal fidelity is becoming another emerging concern in image/video compression. With the recent advances in cross modal translation and generation, in this paper, we propose the cross modal compression~(CMC), a semantic compression framework for visual data, to transform the high redundant visual data~(such as image, video, etc.) into a compact, human-comprehensible domain~(such as text, sketch, semantic map, attributions, etc.), while preserving the semantic. Specifically, we first formulate the CMC problem as a rate-distortion optimization problem. Secondly, we investigate the relationship with the traditional image/video compression and the recent feature compression frameworks, showing the difference between our CMC and these prior frameworks. Then we propose a novel paradigm for CMC to demonstrate its effectiveness. The qualitative and quantitative results show that our proposed CMC can achieve encouraging reconstructed results with an ultrahigh compression ratio, showing better compression performance than the widely used JPEG baseline.