Spatio-Temporal Mixed and Augmented Reality Experience Description for Interactive Playback

TL;DR

The paper introduces MAR-ED, a framework for describing and replaying past experiences in mixed and augmented reality, enabling adaptive, interactive playback that conforms to new environments and user input.

Contribution

It presents a novel descriptive framework with core primitives for semantic, efficient, and interactive representation of experiences in AR/VR environments.

Findings

Enables adaptive, interactive playback of recorded experiences in new environments.

Transforms passive memories into immersive, spatially-integrated group experiences.

Supports applications in training, cultural heritage, and storytelling.

Abstract

We propose the Spatio-Temporal Mixed and Augmented Reality Experience Description (MAR-ED), a novel framework to standardize the representation of past events for interactive and adaptive playback in a user's present physical space. While current spatial media technologies have primarily focused on capturing or replaying content as static assets, often disconnected from the viewer's environment or offering limited interactivity, the means to describe an experience's underlying semantic and interactive structure remains underexplored. We propose a descriptive framework called MAR-ED based on three core primitives: 1) Event Primitives for semantic scene graph representation, 2) Keyframe Primitives for efficient and meaningful data access, and 3) Playback Primitives for user-driven adaptive interactive playback of recorded MAR experience. The proposed flowchart of the three-stage process of the proposed MAR-ED framework transforms a recorded experience into a unique adaptive MAR experience during playback, where its spatio-temporal structure dynamically conforms to a new environment and its narrative can be altered by live user input. Drawing on this framework, personal digital memories and recorded events can evolve beyond passive 2D/3D videos into immersive, spatially-integrated group experiences, opening new paradigms for training, cultural heritage, and interactive storytelling without requiring complex, per-user adaptive rendering.