Improving Landmark Localization with Semi-Supervised Learning

TL;DR

This paper introduces semi-supervised techniques for landmark localization that leverage auxiliary class labels and equivariance principles, significantly improving accuracy with limited labeled data.

Contribution

It proposes a sequential multitasking framework and an unsupervised equivariance-based method for landmark localization with partially annotated datasets.

Findings

Achieved state-of-the-art results on multiple datasets.

Improved landmark detection accuracy with only 5% labeled data.

Demonstrated effectiveness on real-world face and hand datasets.

Abstract

We present two techniques to improve landmark localization in images from partially annotated datasets. Our primary goal is to leverage the common situation where precise landmark locations are only provided for a small data subset, but where class labels for classification or regression tasks related to the landmarks are more abundantly available. First, we propose the framework of sequential multitasking and explore it here through an architecture for landmark localization where training with class labels acts as an auxiliary signal to guide the landmark localization on unlabeled data. A key aspect of our approach is that errors can be backpropagated through a complete landmark localization model. Second, we propose and explore an unsupervised learning technique for landmark localization based on having a model predict equivariant landmarks with respect to transformations applied to the image. We show that these techniques, improve landmark prediction considerably and can learn effective detectors even when only a small fraction of the dataset has landmark labels. We present results on two toy datasets and four real datasets, with hands and faces, and report new state-of-the-art on two datasets in the wild, e.g. with only 5\% of labeled images we outperform previous state-of-the-art trained on the AFLW dataset.