From 2D Document Interactions into Immersive Information Experience: An Example-Based Design by Augmenting Content, Spatializing Placement, Enriching Long-Term Interactions, and Simplifying Content Creations

TL;DR

This research explores transforming traditional 2D document interactions into immersive experiences using VR, MR, voice assistants, and AI, to enhance understanding, interaction, and creation of complex documents.

Contribution

It introduces a comprehensive approach combining multiple emergent technologies to enhance 2D document interaction and creation in immersive environments.

Findings

Enhanced understanding of medical scans and reference images in immersive settings

Improved interaction with instruction documents through spatialization and voice support

Simplified document creation using generative AI techniques

Abstract

Documents serve as a crucial and indispensable medium for everyday workplace tasks. However, understanding, interacting and creating such documents on today's planar interfaces without any intelligent support are challenging due to our natural cognitive constraints on remembering, processing, understanding and interacting with these information. My doctorate research investigates how to bring 2D document interactions into immersive information experience using multiple of today's emergent technologies. With the examples of four specific types of documents -- medical scans, instruction document, self-report diary survey, and reference images for visual artists -- my research demonstrates how to transform such of today's 2D document interactions into an immersive information experience, by augmenting content with virtual reality, spatializing document placements with mixed reality, enriching long-term and continuous interactions with voice assistants, and simplify document creation workflow with generative AI.