Sketch-based Image Retrieval from Millions of Images under Rotation, Translation and Scale Variations

TL;DR

This paper introduces a scalable, rotation, translation, and scale-invariant sketch-based image retrieval method for millions of images, utilizing contour chains, similarity-invariant descriptors, and hierarchical indexing to improve accuracy and efficiency.

Contribution

The paper presents a novel large-scale image retrieval approach that handles similarity transformations and small deformations, outperforming existing methods in accuracy and speed.

Findings

Handles rotation, translation, and scale variations effectively.

Uses hierarchical indexing for fast retrieval from millions of images.

Demonstrates superior performance over existing methods.

Abstract

Proliferation of touch-based devices has made sketch-based image retrieval practical. While many methods exist for sketch-based object detection/image retrieval on small datasets, relatively less work has been done on large (web)-scale image retrieval. In this paper, we present an efficient approach for image retrieval from millions of images based on user-drawn sketches. Unlike existing methods for this problem which are sensitive to even translation or scale variations, our method handles rotation, translation, scale (i.e. a similarity transformation) and small deformations. The object boundaries are represented as chains of connected segments and the database images are pre-processed to obtain such chains that have a high chance of containing the object. This is accomplished using two approaches in this work: a) extracting long chains in contour segment networks and b) extracting boundaries of segmented object proposals. These chains are then represented by similarity-invariant variable length descriptors. Descriptor similarities are computed by a fast Dynamic Programming-based partial matching algorithm. This matching mechanism is used to generate a hierarchical k-medoids based indexing structure for the extracted chains of all database images in an offline process which is used to efficiently retrieve a small set of possible matched images for query chains. Finally, a geometric verification step is employed to test geometric consistency of multiple chain matches to improve results. Qualitative and quantitative results clearly demonstrate superiority of the approach over existing methods.