Through the PRISm: Importance-Aware Scene Graphs for Image Retrieval

TL;DR

PRISm introduces an importance-aware scene graph approach with a novel importance prediction and relational encoding, significantly improving semantic image retrieval accuracy and interpretability.

Contribution

It presents a new multimodal framework combining importance prediction and edge-aware graph neural networks for enhanced semantic image retrieval.

Findings

Achieves superior retrieval performance on benchmark datasets.

Effectively captures key objects and interactions for interpretability.

Outperforms prior methods in semantic alignment with human perception.

Abstract

Accurately retrieving images that are semantically similar remains a fundamental challenge in computer vision, as traditional methods often fail to capture the relational and contextual nuances of a scene. We introduce PRISm (Pruning-based Image Retrieval via Importance Prediction on Semantic Graphs), a multimodal framework that advances image-to-image retrieval through two novel components. First, the Importance Prediction Module identifies and retains the most critical objects and relational triplets within an image while pruning irrelevant elements. Second, the Edge-Aware Graph Neural Network explicitly encodes relational structure and integrates global visual features to produce semantically informed image embeddings. PRISm achieves image retrieval that closely aligns with human perception by explicitly modeling the semantic importance of objects and their interactions, capabilities largely absent in prior approaches. Its architecture effectively combines relational reasoning with visual representation, enabling semantically grounded retrieval. Extensive experiments on benchmark and real-world datasets demonstrate consistently superior top-ranked performance, while qualitative analyses show that PRISm accurately captures key objects and interactions, producing interpretable and semantically meaningful results.