Cross-Modal Scene Semantic Alignment for Image Complexity Assessment

TL;DR

This paper introduces a novel cross-modal scene semantic alignment method for image complexity assessment, leveraging semantic information from text-image pairs to improve alignment with human perception and outperform existing methods.

Contribution

The paper proposes a new cross-modal approach for ICA that aligns scene semantics from text and images, enhancing prediction accuracy over prior single-modality methods.

Findings

Significantly outperforms state-of-the-art ICA methods on multiple datasets.

Demonstrates the effectiveness of cross-modal semantic alignment in capturing image complexity.

Provides publicly available code for reproducibility.

Abstract

Image complexity assessment (ICA) is a challenging task in perceptual evaluation due to the subjective nature of human perception and the inherent semantic diversity in real-world images. Existing ICA methods predominantly rely on hand-crafted or shallow convolutional neural network-based features of a single visual modality, which are insufficient to fully capture the perceived representations closely related to image complexity. Recently, cross-modal scene semantic information has been shown to play a crucial role in various computer vision tasks, particularly those involving perceptual understanding. However, the exploration of cross-modal scene semantic information in the context of ICA remains unaddressed. Therefore, in this paper, we propose a novel ICA method called Cross-Modal Scene Semantic Alignment (CM-SSA), which leverages scene semantic alignment from a cross-modal perspective to enhance ICA performance, enabling complexity predictions to be more consistent with subjective human perception. Specifically, the proposed CM-SSA consists of a complexity regression branch and a scene semantic alignment branch. The complexity regression branch estimates image complexity levels under the guidance of the scene semantic alignment branch, while the scene semantic alignment branch is used to align images with corresponding text prompts that convey rich scene semantic information by pair-wise learning. Extensive experiments on several ICA datasets demonstrate that the proposed CM-SSA significantly outperforms state-of-the-art approaches. Codes are available at https://github.com/XQ2K/First-Cross-Model-ICA.