Towards Practicality of Sketch-Based Visual Understanding

TL;DR

This paper reviews the use of sketches in visual understanding, highlighting their potential and challenges, and discusses progress towards making sketch-based applications more practical and accessible.

Contribution

It analyzes the current state of sketch-based visual understanding and identifies key challenges and opportunities for enhancing practicality and user adoption.

Findings

Sketches enable fine-grained visual control across diverse tasks.

Subjectivity and skill limitations hinder sketch-based applications.

Collecting sketch-photo data remains a significant bottleneck.

Abstract

Sketches have been used to conceptualise and depict visual objects from pre-historic times. Sketch research has flourished in the past decade, particularly with the proliferation of touchscreen devices. Much of the utilisation of sketch has been anchored around the fact that it can be used to delineate visual concepts universally irrespective of age, race, language, or demography. The fine-grained interactive nature of sketches facilitates the application of sketches to various visual understanding tasks, like image retrieval, image-generation or editing, segmentation, 3D-shape modelling etc. However, sketches are highly abstract and subjective based on the perception of individuals. Although most agree that sketches provide fine-grained control to the user to depict a visual object, many consider sketching a tedious process due to their limited sketching skills compared to other query/support modalities like text/tags. Furthermore, collecting fine-grained sketch-photo association is a significant bottleneck to commercialising sketch applications. Therefore, this thesis aims to progress sketch-based visual understanding towards more practicality.