Deep Semantic Parsing of Freehand Sketches with Homogeneous Transformation, Soft-Weighted Loss, and Staged Learning

TL;DR

This paper introduces a deep learning framework for part-level semantic parsing of freehand sketches, utilizing a homogeneous transformation for domain adaptation, a soft-weighted loss for better training, and staged learning for improved accuracy, achieving state-of-the-art results.

Contribution

The paper presents a novel homogeneous transformation method for domain adaptation, a soft-weighted loss function, and a staged learning strategy, collectively advancing freehand sketch semantic parsing.

Findings

Homogeneous transformation reduces domain gap between real images and sketches.

Soft-weighted loss improves handling of ambiguous boundaries and class imbalance.

The integrated framework achieves state-of-the-art performance on SketchParse dataset.

Abstract

In this paper, we propose a novel deep framework for part-level semantic parsing of freehand sketches, which makes three main contributions that are experimentally shown to have substantial practical merit. First, we propose a homogeneous transformation method to address the problem of domain adaptation. For the task of sketch parsing, there is no available data of labeled freehand sketches that can be directly used for model training. An alternative solution is to learn from datasets of real image parsing, while the domain adaptation is an inevitable problem. Unlike existing methods that utilize the edge maps of real images to approximate freehand sketches, the proposed homogeneous transformation method transforms the data from domains of real images and freehand sketches into a homogeneous space to minimize the semantic gap. Second, we design a soft-weighted loss function as guidance for the training process, which gives attention to both the ambiguous label boundary and class imbalance. Third, we present a staged learning strategy to improve the parsing performance of the trained model, which takes advantage of the shared information and specific characteristic from different sketch categories. Extensive experimental results demonstrate the effectiveness of the above three methods. Specifically, to evaluate the generalization ability of our homogeneous transformation method, additional experiments for the task of sketch-based image retrieval are conducted on the QMUL FG-SBIR dataset. Finally, by integrating the proposed three methods into a unified framework of deep semantic sketch parsing (DeepSSP), we achieve the state-of-the-art on the public SketchParse dataset.