Deep Zero-Shot Learning for Scene Sketch

TL;DR

This paper introduces a deep embedding model for scene sketch zero-shot learning, addressing the challenges of semantic gap and complex feature representation, by fusing multi-modal knowledge and using attention mechanisms.

Contribution

It proposes an augmented semantic vector for domain alignment and a novel distance metric, advancing scene sketch zero-shot learning techniques.

Findings

Enhanced domain alignment with multi-modal semantic fusion

Improved sketch feature learning via attention mechanisms

Superior performance demonstrated through extensive experiments

Abstract

We introduce a novel problem of scene sketch zero-shot learning (SSZSL), which is a challenging task, since (i) different from photo, the gap between common semantic domain (e.g., word vector) and sketch is too huge to exploit common semantic knowledge as the bridge for knowledge transfer, and (ii) compared with single-object sketch, more expressive feature representation for scene sketch is required to accommodate its high-level of abstraction and complexity. To overcome these challenges, we propose a deep embedding model for scene sketch zero-shot learning. In particular, we propose the augmented semantic vector to conduct domain alignment by fusing multi-modal semantic knowledge (e.g., cartoon image, natural image, text description), and adopt attention-based network for scene sketch feature learning. Moreover, we propose a novel distance metric to improve the similarity measure during testing. Extensive experiments and ablation studies demonstrate the benefit of our sketch-specific design.