SketchXAI: A First Look at Explainability for Human Sketches

TL;DR

This paper pioneers the integration of human sketches into explainable AI, introducing a sketch-specific explainability task and a novel encoder that improves recognition accuracy while enabling stroke location inversion for interpretability.

Contribution

It introduces the first explainability framework for sketches, including a stroke location inversion task and a specialized encoder that enhances recognition accuracy.

Findings

Successful implementation of stroke location inversion (SLI) as an explainability task.

The sketch encoder achieves the best recognition accuracy with minimal parameters.

Qualitative results demonstrate interpretability of the SLI process.

Abstract

This paper, for the very first time, introduces human sketches to the landscape of XAI (Explainable Artificial Intelligence). We argue that sketch as a ``human-centred'' data form, represents a natural interface to study explainability. We focus on cultivating sketch-specific explainability designs. This starts by identifying strokes as a unique building block that offers a degree of flexibility in object construction and manipulation impossible in photos. Following this, we design a simple explainability-friendly sketch encoder that accommodates the intrinsic properties of strokes: shape, location, and order. We then move on to define the first ever XAI task for sketch, that of stroke location inversion SLI. Just as we have heat maps for photos, and correlation matrices for text, SLI offers an explainability angle to sketch in terms of asking a network how well it can recover stroke locations of an unseen sketch. We offer qualitative results for readers to interpret as snapshots of the SLI process in the paper, and as GIFs on the project page. A minor but interesting note is that thanks to its sketch-specific design, our sketch encoder also yields the best sketch recognition accuracy to date while having the smallest number of parameters. The code is available at \url{https://sketchxai.github.io}.