Real-Time Shape Tracking of Facial Landmarks

TL;DR

This paper presents a real-time facial landmark shape tracking method that combines deep learning and point tracking to improve accuracy and robustness in virtual makeup applications.

Contribution

It introduces a hybrid approach using SegNet for precise landmark detection and KLT for tracking, addressing point loss and computational efficiency issues.

Findings

Accurate real-time shape tracking demonstrated in experiments.

Effective combination reduces point loss problems.

Maintains real-time performance in virtual makeup scenarios.

Abstract

Detection of facial landmarks and accurate tracking of their shape are essential in real-time virtual makeup applications, where users can see the makeups effect by moving their face in different directions. Typical face tracking techniques detect diverse facial landmarks and track them using a point tracker such as the Kanade-Lucas-Tomasi (KLT) point tracker. Typically, 5 or 64 points are used for tracking a face. Even though these points are sufficient to track the approximate locations of facial landmarks, they are not sufficient to track the exact shape of facial landmarks. In this paper, we propose a method that can track the exact shape of facial landmarks in real-time by combining a deep learning technique and a point tracker. We detect facial landmarks accurately using SegNet, which performs semantic segmentation based on deep learning. Edge points of detected landmarks are tracked using the KLT point tracker. In spite of its popularity, the KLT point tracker suffers from the point loss problem. We solve this problem by executing SegNet periodically to calculate the shape of facial landmarks. That is, by combining the two techniques, we can avoid the computational overhead of SegNet for real-time shape tracking and the point loss problem of the KLT point tracker. We performed several experiments to evaluate the performance of our method and report some of the results herein.