SketchRef: a Multi-Task Evaluation Benchmark for Sketch Synthesis

TL;DR

SketchRef is a comprehensive benchmark for evaluating sketch synthesis methods, introducing new tasks, metrics, and a large dataset to standardize and advance research in the field.

Contribution

It presents the first multi-task benchmark for sketch synthesis, including novel evaluation metrics and a large annotated dataset across multiple domains.

Findings

Existing methods show varied strengths and weaknesses on the benchmark.

The recognizability-simplicity trade-off can be quantitatively assessed.

The benchmark enables standardized evaluation of sketch synthesis algorithms.

Abstract

Sketching is a powerful artistic technique for capturing essential visual information about real-world objects and has increasingly attracted attention in image synthesis research. However, the field lacks a unified benchmark to evaluate the performance of various synthesis methods. To address this, we propose SketchRef, the first comprehensive multi-task evaluation benchmark for sketch synthesis. SketchRef fully leverages the shared characteristics between sketches and reference photos. It introduces two primary tasks: category prediction and structural consistency estimation, the latter being largely overlooked in previous studies. These tasks are further divided into five sub-tasks across four domains: animals, common things, human body, and faces. Recognizing the inherent trade-off between recognizability and simplicity in sketches, we are the first to quantify this balance by introducing a recognizability calculation method constrained by simplicity, mRS, ensuring fair and meaningful evaluations. To validate our approach, we collected 7,920 responses from art enthusiasts, confirming the effectiveness of our proposed evaluation metrics. Additionally, we evaluate the performance of existing sketch synthesis methods on our benchmark, highlighting their strengths and weaknesses. We hope this study establishes a standardized benchmark and offers valuable insights for advancing sketch synthesis algorithms.