HoloFace: Augmenting Human-to-Human Interactions on HoloLens

TL;DR

HoloFace is an open-source framework that enables face alignment, head pose estimation, and facial attribute retrieval on HoloLens to enhance human interactions through face augmentation.

Contribution

It introduces a flexible framework with interchangeable face alignment methods and demonstrates practical face augmentation applications on HoloLens.

Findings

Real-time face and head pose estimation on HoloLens

Effective facial attribute retrieval for augmentation

Open-source implementation for broader use

Abstract

We present HoloFace, an open-source framework for face alignment, head pose estimation and facial attribute retrieval for Microsoft HoloLens. HoloFace implements two state-of-the-art face alignment methods which can be used interchangeably: one running locally and one running on a remote backend. Head pose estimation is accomplished by fitting a deformable 3D model to the landmarks localized using face alignment. The head pose provides both the rotation of the head and a position in the world space. The parameters of the fitted 3D face model provide estimates of facial attributes such as mouth opening or smile. Together the above information can be used to augment the faces of people seen by the HoloLens user, and thus their interaction. Potential usage scenarios include facial recognition, emotion recognition, eye gaze tracking and many others. We demonstrate the capabilities of our framework by augmenting the faces of people seen through the HoloLens with various objects and animations.