Semantic Similarity Score for Measuring Visual Similarity at Semantic Level

TL;DR

This paper introduces SeSS, a semantic similarity metric based on scene graph matching, to evaluate semantic-level image similarity, addressing limitations of traditional pixel-based metrics in semantic communication systems.

Contribution

The paper proposes a novel semantic similarity score, SeSS, utilizing scene graph generation and matching, with fine-tuning based on human-annotated data to better reflect semantic perception.

Findings

SeSS effectively measures semantic differences in images transmitted by various communication systems.

SeSS correlates well with human semantic perception in diverse scenarios.

The metric outperforms traditional pixel-based similarity measures in semantic evaluation.

Abstract

Semantic communication, as a revolutionary communication architecture, is considered a promising novel communication paradigm. Unlike traditional symbol-based error-free communication systems, semantic-based visual communication systems extract, compress, transmit, and reconstruct images at the semantic level. However, widely used image similarity evaluation metrics, whether pixel-based MSE or PSNR or structure-based MS-SSIM, struggle to accurately measure the loss of semantic-level information of the source during system transmission. This presents challenges in evaluating the performance of visual semantic communication systems, especially when comparing them with traditional communication systems. To address this, we propose a semantic evaluation metric -- SeSS (Semantic Similarity Score), based on Scene Graph Generation and graph matching, which shifts the similarity scores between images into semantic-level graph matching scores. Meanwhile, semantic similarity scores for tens of thousands of image pairs are manually annotated to fine-tune the hyperparameters in the graph matching algorithm, aligning the metric more closely with human semantic perception. The performance of the SeSS is tested on different datasets, including (1)images transmitted by traditional and semantic communication systems at different compression rates, (2)images transmitted by traditional and semantic communication systems at different signal-to-noise ratios, (3)images generated by large-scale model with different noise levels introduced, and (4)cases of images subjected to certain special transformations. The experiments demonstrate the effectiveness of SeSS, indicating that the metric can measure the semantic-level differences in semantic-level information of images and can be used for evaluation in visual semantic communication systems.