Teachable Reality: Prototyping Tangible Augmented Reality with Everyday Objects by Leveraging Interactive Machine Teaching

TL;DR

Teachable Reality is an AR prototyping tool that uses vision-based machine teaching to enable users to create interactive tangible AR applications with everyday objects without programming.

Contribution

It introduces a flexible, customizable AR prototyping approach leveraging interactive machine teaching and gesture recognition, overcoming limitations of marker-based AR methods.

Findings

Lowered barrier to AR prototyping for users.

Enabled creation of diverse AR prototypes including deformable interfaces.

User and expert feedback confirmed increased flexibility and usability.

Abstract

This paper introduces Teachable Reality, an augmented reality (AR) prototyping tool for creating interactive tangible AR applications with arbitrary everyday objects. Teachable Reality leverages vision-based interactive machine teaching (e.g., Teachable Machine), which captures real-world interactions for AR prototyping. It identifies the user-defined tangible and gestural interactions using an on-demand computer vision model. Based on this, the user can easily create functional AR prototypes without programming, enabled by a trigger-action authoring interface. Therefore, our approach allows the flexibility, customizability, and generalizability of tangible AR applications that can address the limitation of current marker-based approaches. We explore the design space and demonstrate various AR prototypes, which include tangible and deformable interfaces, context-aware assistants, and body-driven AR applications. The results of our user study and expert interviews confirm that our approach can lower the barrier to creating functional AR prototypes while also allowing flexible and general-purpose prototyping experiences.