Egocentric Human Segmentation for Mixed Reality

TL;DR

This paper presents a semi-synthetic dataset and a fast deep learning method for segmenting human body parts from egocentric videos, aiming to improve realism in virtual environments.

Contribution

It introduces a large semi-synthetic dataset and a real-time capable semantic segmentation network based on ThunderNet architecture for egocentric human segmentation.

Findings

Achieved segmentation in 16 ms for 720x720 images.

Created a dataset with over 15,000 annotated images.

Enhanced realism in virtual avatars.

Abstract

The objective of this work is to segment human body parts from egocentric video using semantic segmentation networks. Our contribution is two-fold: i) we create a semi-synthetic dataset composed of more than 15, 000 realistic images and associated pixel-wise labels of egocentric human body parts, such as arms or legs including different demographic factors; ii) building upon the ThunderNet architecture, we implement a deep learning semantic segmentation algorithm that is able to perform beyond real-time requirements (16 ms for 720 x 720 images). It is believed that this method will enhance sense of presence of Virtual Environments and will constitute a more realistic solution to the standard virtual avatars.