Region-Based Image Retrieval Revisited

TL;DR

This paper enhances region-based image retrieval by integrating semantic object specifications with deep learning and intuitive spatial relationship recommendations, enabling more accurate and user-friendly image searches.

Contribution

It introduces a multitask CNN for multi-aspect object specification and a recommendation system for spatial relationships, improving RBIR's semantic and spatial querying capabilities.

Findings

Deep learning-based features improve object characterization.

Spatial relationship recommendations enhance query accuracy.

Fast indexing and re-ranking enable real-time retrieval.

Abstract

Region-based image retrieval (RBIR) technique is revisited. In early attempts at RBIR in the late 90s, researchers found many ways to specify region-based queries and spatial relationships; however, the way to characterize the regions, such as by using color histograms, were very poor at that time. Here, we revisit RBIR by incorporating semantic specification of objects and intuitive specification of spatial relationships. Our contributions are the following. First, to support multiple aspects of semantic object specification (category, instance, and attribute), we propose a multitask CNN feature that allows us to use deep learning technique and to jointly handle multi-aspect object specification. Second, to help users specify spatial relationships among objects in an intuitive way, we propose recommendation techniques of spatial relationships. In particular, by mining the search results, a system can recommend feasible spatial relationships among the objects. The system also can recommend likely spatial relationships by assigned object category names based on language prior. Moreover, object-level inverted indexing supports very fast shortlist generation, and re-ranking based on spatial constraints provides users with instant RBIR experiences.