Video2MR: Automatically Generating Mixed Reality 3D Instructions by Augmenting Extracted Motion from 2D Videos

TL;DR

Video2MR automatically converts 2D instructional videos into immersive mixed reality 3D experiences, enhancing physical training by providing engaging, customizable, and easily generated MR instructions using AI-enabled motion capture and augmentation techniques.

Contribution

This work introduces a novel system that transforms online 2D instructional videos into MR 3D avatars with augmented motion, reducing time and cost compared to traditional methods.

Findings

Enhanced engagement and playfulness in learning experiences

Successful implementation on Hololens 2 and Azure Kinect

Versatile use cases including yoga, dancing, and sports

Abstract

This paper introduces Video2MR, a mixed reality system that automatically generates 3D sports and exercise instructions from 2D videos. Mixed reality instructions have great potential for physical training, but existing works require substantial time and cost to create these 3D experiences. Video2MR overcomes this limitation by transforming arbitrary instructional videos available online into MR 3D avatars with AI-enabled motion capture (DeepMotion). Then, it automatically enhances the avatar motion through the following augmentation techniques: 1) contrasting and highlighting differences between the user and avatar postures, 2) visualizing key trajectories and movements of specific body parts, 3) manipulation of time and speed using body motion, and 4) spatially repositioning avatars for different perspectives. Developed on Hololens 2 and Azure Kinect, we showcase various use cases, including yoga, dancing, soccer, tennis, and other physical exercises. The study results confirm that Video2MR provides more engaging and playful learning experiences, compared to existing 2D video instructions.