How to Evaluate Semantic Communications for Images with ViTScore Metric?

TL;DR

This paper introduces ViTScore, a novel metric inspired by NLP's BERTScore, designed to evaluate semantic similarity between images, addressing limitations of traditional pixel-based metrics in semantic communication systems.

Contribution

The paper proposes ViTScore, a new image semantic similarity metric with proven properties, and demonstrates its effectiveness over existing metrics in various semantic communication scenarios.

Findings

ViTScore outperforms PSNR, MS-SSIM, and LPIPS in semantic similarity evaluation.

ViTScore is robust against semantic attacks like GAN-based image inversions.

The metric is theoretically justified with properties like symmetry, boundedness, and normalization.

Abstract

Semantic communications (SC) have been expected to be a new paradigm shifting to catalyze the next generation communication, whose main concerns shift from accurate bit transmission to effective semantic information exchange in communications. However, the previous and widely-used metrics for images are not applicable to evaluate the image semantic similarity in SC. Classical metrics to measure the similarity between two images usually rely on the pixel level or the structural level, such as the PSNR and the MS-SSIM. Straightforwardly using some tailored metrics based on deep-learning methods in CV community, such as the LPIPS, is infeasible for SC. To tackle this, inspired by BERTScore in NLP community, we propose a novel metric for evaluating image semantic similarity, named Vision Transformer Score (ViTScore). We prove theoretically that ViTScore has 3 important properties, including symmetry, boundedness, and normalization, which make ViTScore convenient and intuitive for image measurement. To evaluate the performance of ViTScore, we compare ViTScore with 3 typical metrics (PSNR, MS-SSIM, and LPIPS) through 4 classes of experiments: (i) correlation with BERTScore through evaluation of image caption downstream CV task, (ii) evaluation in classical image communications, (iii) evaluation in image semantic communication systems, and (iv) evaluation in image semantic communication systems with semantic attack. Experimental results demonstrate that ViTScore is robust and efficient in evaluating the semantic similarity of images. Particularly, ViTScore outperforms the other 3 typical metrics in evaluating the image semantic changes by semantic attack, such as image inverse with Generative Adversarial Networks (GANs). This indicates that ViTScore is an effective performance metric when deployed in SC scenarios.