Immersive Experiences and XR: A Game Engine or Multimedia Streaming Problem?

TL;DR

This paper explores the architectural challenges of XR technology, emphasizing the integration of game engineering and multimedia streaming to support complex, multi-user immersive experiences.

Contribution

It provides a comprehensive overview of the functional components and architecture needed for XR systems, highlighting the importance of standardization and interoperability.

Findings

Identifies new metadata and media orchestration requirements for XR.

Proposes functional blocks for XR system architectures.

Discusses standardization efforts by Khronos, MPEG, and 3GPP.

Abstract

Recent improvements in Extended Reality (XR) technology created an increase in XR products and solutions in the industry, while raising new requirements for new or improved architectural concepts. This need can be particularly complex as XR applications often relate both to 3D geometric rendering and multimedia paradigms. This paper outlines the main concepts relevant to XR, both from a game engineering and multimedia streaming system perspective. XR requires new metadata and media/game orchestration to allow complex interaction between users, objects, and (volumetric) multimedia content, which also results in new requirements on synchronisation (i.e., for global object state and positioning). Furthermore, the paper presents the functional blocks needed in new XR system architectures and how they will glue both (game and media) spaces together. The discussion of functional components and architecture relates to the ongoing activities in relevant standardisation bodies like Khronos, MPEG, and 3GPP. To make XR successful on the long term, the industry needs to agree on interoperable solutions and how to merge both game and media paradigms to allow complex multi-user XR applications.