Drawing with AI -- Exploring Collaborative Inking Experiences Based on Mid-air Pointing and Reinforcement Learning

TL;DR

This paper presents a novel AI-enhanced digital drawing tool using mid-air pointing and reinforcement learning, enabling intuitive collaborative drawing experiences across diverse users and display sizes.

Contribution

It introduces a new interactive drawing system combining mid-air pointing with reinforcement learning and evaluates its usability through a qualitative user study.

Findings

Users found the system engaging and intuitive.

The reinforcement learning component adapted well to user input.

The system supported collaborative and accessible drawing experiences.

Abstract

Digitalization is changing the nature of tools and materials, which are used in artistic practices in professional and non-professional settings. For example, today it is common that even children express their ideas and explore their creativity by drawing on tablets as digital canvases. While there are many software-based tools, which resemble traditional tools, such as various forms of virtual brushes, erasers, etc. in contrast to traditional materials there is potential in augmenting software-based tools and digital canvases with artificial intelligence. Curious about how it would feel to interact with a digital canvas, which would be in contrast to a traditional canvas dynamic, responsive, and potentially able to continuously adapt to its user's input, we developed a drawing application and conducted a qualitative study with 14 users. In this paper, we describe details of our design process, which lead up to using a k-armed bandit as a simple form of reinforcement learning and a LeapMotion sensor to allow people from all walks of like, old and young to draw on pervasive displays, small and large, positioned near or far.