SnapNCode: An Integrated Development Environment for Programming Physical Objects Interactions

TL;DR

SnapNCode is an innovative IDE designed for spatial programming that allows direct interaction with physical objects through live video and augmented code snippets, bridging the gap between real-world objects and digital code.

Contribution

It introduces a novel spatial programming IDE that integrates physical object states into code via live video and augmented interactions, addressing limitations of traditional IDEs.

Findings

Participants found SnapNCode easy to use.

System shows promise for casual and workflow integration.

User feedback indicates high usability and potential.

Abstract

Spatial computing technologies have the potential to revolutionize how we interact with the world around us. However, most modern integrated development environments (IDEs) have not fully adapted to this paradigm shift. For example, physical 3D objects in the real world are still represented as 2D text variables in code, creating a significant perceptual distance between these representations. In response to this challenge, we introduce SnapNCode, a novel IDE for spatial programming. SnapNCode enables programmers to capture various states of physical objects through live video streams from cameras and directly insert these visual representations into their code. Moreover, users can augment physical objects by attaching code snippets onto objects, which are opportunistically triggered when observed by cameras. We conducted a user study (N=12) to assess the usability of SnapNCode. Feedback from participants indicates that the system is easy-to-use and holds promise for daily casual uses and integration into a broader range of workflows.