NAPA: Neural Art Human Pose Amplifier

TL;DR

This paper introduces NAPA, an end-to-end neural style transfer-based system for artistic human pose estimation, leveraging style transfer, pseudo 3D labels, and self-supervision to improve pose regression in artistic images.

Contribution

It proposes a novel neural style transfer approach combined with pose regression and pseudo 3D labels, enhancing pose estimation in artistic images.

Findings

Promising pose regression results on artistic test set.

Effective use of style transfer and pseudo 3D labels for pose estimation.

Model generalizes to real-world human datasets.

Abstract

This is the project report for CSCI-GA.2271-001. We target human pose estimation in artistic images. For this goal, we design an end-to-end system that uses neural style transfer for pose regression. We collect a 277-style set for arbitrary style transfer and build an artistic 281-image test set. We directly run pose regression on the test set and show promising results. For pose regression, we propose a 2d-induced bone map from which pose is lifted. To help such a lifting, we additionally annotate the pseudo 3d labels of the full in-the-wild MPII dataset. Further, we append another style transfer as self supervision to improve 2d. We perform extensive ablation studies to analyze the introduced features. We also compare end-to-end with per-style training and allude to the tradeoff between style transfer and pose regression. Lastly, we generalize our model to the real-world human dataset and show its potentiality as a generic pose model. We explain the theoretical foundation in Appendix. We release code at https://github.com/strawberryfg/NAPA-NST-HPE, data, and video.