Sketched Reality: Sketching Bi-Directional Interactions Between Virtual and Physical Worlds with AR and Actuated Tangible UI

TL;DR

This paper presents Sketched Reality, a novel system combining AR sketching and actuated tangible interfaces to enable bidirectional interaction between virtual sketches and physical objects, expanding possibilities for education, gaming, and robot programming.

Contribution

It introduces a new bi-directional sketching interaction method that couples virtual AR sketches with physical actuated robots, creating seamless virtual-physical coupling.

Findings

Demonstrated virtual sketches can control physical robots in real-time.

Enabled new applications like tangible physics education and robot programming.

Showcased seamless interaction between AR and physical objects.

Abstract

This paper introduces Sketched Reality, an approach that combines AR sketching and actuated tangible user interfaces (TUI) for bidirectional sketching interaction. Bi-directional sketching enables virtual sketches and physical objects to "affect" each other through physical actuation and digital computation. In the existing AR sketching, the relationship between virtual and physical worlds is only one-directional -- while physical interaction can affect virtual sketches, virtual sketches have no return effect on the physical objects or environment. In contrast, bi-directional sketching interaction allows the seamless coupling between sketches and actuated TUIs. In this paper, we employ tabletop-size small robots (Sony Toio) and an iPad-based AR sketching tool to demonstrate the concept. In our system, virtual sketches drawn and simulated on an iPad (e.g., lines, walls, pendulums, and springs) can move, actuate, collide, and constrain physical Toio robots, as if virtual sketches and the physical objects exist in the same space through seamless coupling between AR and robot motion. This paper contributes a set of novel interactions and a design space of bi-directional AR sketching. We demonstrate a series of potential applications, such as tangible physics education, explorable mechanism, tangible gaming for children, and in-situ robot programming via sketching.