Towards an End-to-End System for 3D Tracking of Physical Objects in Virtual Immersive Environments

TL;DR

This paper presents an end-to-end, plug-and-play system for real-time 3D physical object tracking in VR using fiducial markers, enabling immersive training without complex hardware.

Contribution

It introduces a novel marker-based tracking system combining existing and original fiducial methods for efficient object tracking in VR environments.

Findings

System achieves fast and reliable object detection across various tag sizes and distances.

Evaluation shows the system's effectiveness with different camera devices.

Provides a complete solution for marker-based object position mapping in VR.

Abstract

This work aims to establish an end-to-end system for tracking of physical 3D objects for virtual reality (VR) applications. We focus on training applications requiring real-time tracking of the position of small physical objects and their reflection in VR space. Out goal is to perform object tracking in a "plug and play" manner, without using complex systems with quite large tracking devices or manually implementing object tracking. We therefore propose a system for object tracking via fiducial markers alongside a software harness, to enable fast and efficient designation of objects to be tracked and data streaming solution for end-use applications. The system utilizes AruCo, AprilTag and an original Colored Control Points based fiducial system. It allows for easy tag detection and use of object position data, which are crucial for immersive training environments based on VR and eXtended Reality (XR). We evaluate various tag sizes, detection distances, and different camera devices against the theoretical limits. In effect, we create a complete solution for implementing marker-based, real-to-virtual object position mapping for various applications.