On Training Sketch Recognizers for New Domains

TL;DR

This paper investigates the challenges of training sketch recognizers for new domains, emphasizing data collection validity, dataset size, and the performance gap between deep learning and traditional methods in realistic, small-data scenarios.

Contribution

It highlights the importance of ecological validity and dataset size in sketch recognition, and compares deep learning with traditional methods under small-data constraints.

Findings

Deep learning methods suffer from dataset shift in realistic scenarios.

Standard data augmentation and transfer learning are insufficient for small datasets.

Traditional classifiers outperform deep learning when data is scarce.

Abstract

Sketch recognition algorithms are engineered and evaluated using publicly available datasets contributed by the sketch recognition community over the years. While existing datasets contain sketches of a limited set of generic objects, each new domain inevitably requires collecting new data for training domain specific recognizers. This gives rise to two fundamental concerns: First, will the data collection protocol yield ecologically valid data? Second, will the amount of collected data suffice to train sufficiently accurate classifiers? In this paper, we draw attention to these two concerns. We show that the ecological validity of the data collection protocol and the ability to accommodate small datasets are significant factors impacting recognizer accuracy in realistic scenarios. More specifically, using sketch-based gaming as a use case, we show that deep learning methods, as well as more traditional methods, suffer significantly from dataset shift. Furthermore, we demonstrate that in realistic scenarios where data is scarce and expensive, standard measures taken for adapting deep learners to small datasets fall short of comparing favorably with alternatives. Although transfer learning, and extensive data augmentation help deep learners, they still perform significantly worse compared to standard setups (e.g., SVMs and GBMs with standard feature representations). We pose learning from small datasets as a key problem for the deep sketch recognition field, one which has been ignored in the bulk of the existing literature.