Query by Semantic Sketch
Luca Rossetto, Ralph Gasser, Heiko Schuldt

TL;DR
This paper introduces a novel sketch-based retrieval method that allows users to query image and video collections by drawing concept maps, capturing semantic labels, their spatial relations, and context for more accurate multimedia search.
Contribution
It presents a new retrieval approach combining sketch-based input with semantic and spatial understanding, integrated into the vitrivr engine for improved multimedia search.
Findings
Effective retrieval of images and videos using semantic sketches.
Captures semantic concepts, relations, and spatial context.
Demonstrated efficiency in large multimedia collections.
Abstract
Sketch-based query formulation is very common in image and video retrieval as these techniques often complement textual retrieval methods that are based on either manual or machine generated annotations. In this paper, we present a retrieval approach that allows to query visual media collections by sketching concept maps, thereby merging sketch-based retrieval with the search for semantic labels. Users can draw a spatial distribution of different concept labels, such as "sky", "sea" or "person" and then use these sketches to find images or video scenes that exhibit a similar distribution of these concepts. Hence, this approach does not only take the semantic concepts themselves into account, but also their semantic relations as well as their spatial context. The efficient vector representation enables efficient retrieval even in large multimedia collections. We have integrated the…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Image and Video Retrieval Techniques · Image Retrieval and Classification Techniques · Multimodal Machine Learning Applications
