Reasoning Structural Relation for Occlusion-Robust Facial Landmark Localization

TL;DR

This paper introduces a structural relation network with a hierarchical module to improve facial landmark localization under occlusion, leveraging shape constraints and occlusion synthesis for robustness.

Contribution

It proposes a hierarchical structural relation module and occlusion synthesis method to enhance landmark localization accuracy in occluded faces.

Findings

Achieves superior performance on occluded face datasets.

Effectively models spatial relations to reduce occlusion impact.

Enhances robustness through occlusion data augmentation.

Abstract

In facial landmark localization tasks, various occlusions heavily degrade the localization accuracy due to the partial observability of facial features. This paper proposes a structural relation network (SRN) for occlusion-robust landmark localization. Unlike most existing methods that simply exploit the shape constraint, the proposed SRN aims to capture the structural relations among different facial components. These relations can be considered a more powerful shape constraint against occlusion. To achieve this, a hierarchical structural relation module (HSRM) is designed to hierarchically reason the structural relations that represent both long- and short-distance spatial dependencies. Compared with existing network architectures, HSRM can efficiently model the spatial relations by leveraging its geometry-aware network architecture, which reduces the semantic ambiguity caused by occlusion. Moreover, the SRN augments the training data by synthesizing occluded faces. To further extend our SRN for occluded video data, we formulate the occluded face synthesis as a Markov decision process (MDP). Specifically, it plans the movement of the dynamic occlusion based on an accumulated reward associated with the performance degradation of the pre-trained SRN. This procedure augments hard samples for robust facial landmark tracking. Extensive experimental results indicate that the proposed method achieves outstanding performance on occluded and masked faces. Code is available at https://github.com/zhuccly/SRN.