Attention Mesh: High-fidelity Face Mesh Prediction in Real-time

TL;DR

Attention Mesh is a fast, lightweight neural network architecture for real-time 3D face mesh prediction on mobile devices, enabling high-accuracy facial landmark applications like AR makeup and eye tracking.

Contribution

It introduces a unified, attention-based network architecture that matches the accuracy of multi-stage methods while being 30% faster for real-time face mesh prediction.

Findings

Runs at over 50 FPS on a Pixel 2 phone

Achieves high accuracy comparable to multi-stage approaches

Enables real-time AR applications

Abstract

We present Attention Mesh, a lightweight architecture for 3D face mesh prediction that uses attention to semantically meaningful regions. Our neural network is designed for real-time on-device inference and runs at over 50 FPS on a Pixel 2 phone. Our solution enables applications like AR makeup, eye tracking and AR puppeteering that rely on highly accurate landmarks for eye and lips regions. Our main contribution is a unified network architecture that achieves the same accuracy on facial landmarks as a multi-stage cascaded approach, while being 30 percent faster.