Leveraging Unlabeled Data for Sketch-based Understanding

TL;DR

This paper explores leveraging unlabeled data to enhance sketch-based understanding by evaluating semi-supervised models and introducing sketch-BYOL, which improves retrieval performance for known and unknown categories.

Contribution

The study introduces sketch-BYOL, a novel self-supervised approach for sketch understanding that outperforms existing methods and demonstrates benefits for various tasks.

Findings

Sketch-BYOL outperforms other self-supervised methods.

Unlabeled data improves sketch retrieval accuracy.

The approach benefits multiple downstream tasks.

Abstract

Sketch-based understanding is a critical component of human cognitive learning and is a primitive communication means between humans. This topic has recently attracted the interest of the computer vision community as sketching represents a powerful tool to express static objects and dynamic scenes. Unfortunately, despite its broad application domains, the current sketch-based models strongly rely on labels for supervised training, ignoring knowledge from unlabeled data, thus limiting the underlying generalization and the applicability. Therefore, we present a study about the use of unlabeled data to improve a sketch-based model. To this end, we evaluate variations of VAE and semi-supervised VAE, and present an extension of BYOL to deal with sketches. Our results show the superiority of sketch-BYOL, which outperforms other self-supervised approaches increasing the retrieval performance for known and unknown categories. Furthermore, we show how other tasks can benefit from our proposal.