ITEM: Immersive Telepresence for Entertainment and Meetings - A Practical Approach

TL;DR

This paper introduces ITEM, an immersive telepresence system that creates a shared virtual space for natural interaction in entertainment and meetings, emphasizing scalability, user experience, and real-time performance.

Contribution

The paper presents a practical, scalable immersive telepresence system with novel technologies like object-based video coding and spatialized audio, optimized for real-time user experience.

Findings

System operates reliably in real time

Minimal setup required for end users

Supports diverse teleimmersive applications

Abstract

This paper presents an Immersive Telepresence system for Entertainment and Meetings (ITEM). The system aims to provide a radically new video communication experience by seamlessly merging participants into the same virtual space to allow a natural interaction among them and shared collaborative contents. With the goal to make a scalable, flexible system for various business solutions as well as easily accessible by massive consumers, we address the challenges in the whole pipeline of media processing, communication, and displaying in our design and realization of such a system. Particularly, in this paper we focus on the system aspects that maximize the end-user experience, optimize the system and network resources, and enable various teleimmersive application scenarios. In addition, we also present a few key technologies, i.e. fast object-based video coding for real world data and spatialized audio capture and 3D sound localization for group teleconferencing. Our effort is to investigate and optimize the key system components and provide an efficient end-to-end optimization and integration by considering user needs and preferences. Extensive experiments show the developed system runs reliably and comfortably in real time with a minimal setup requirement (e.g. a webcam and/or a depth camera, an optional microphone array, a laptop/desktop connected to the public Internet) for teleimmersive communication. With such a really minimal deployment requirement, we present a variety of interesting applications and user experiences created by ITEM.