Unified 3D Mesh Recovery of Humans and Animals by Learning Animal Exercise

TL;DR

This paper introduces a unified end-to-end model for 3D mesh recovery of humans and animals, leveraging morphological similarities and weak supervision to enable multi-task learning across heterogeneous datasets.

Contribution

It proposes a novel multi-task framework that exploits morphological similarities via semantic sub-keypoints and class-sensitive regularization for joint human and animal mesh recovery.

Findings

Outperforms recent uni-modal models on multiple datasets.

Uses a compact model architecture.

Effectively leverages heterogeneous datasets for multi-task learning.

Abstract

We propose an end-to-end unified 3D mesh recovery of humans and quadruped animals trained in a weakly-supervised way. Unlike recent work focusing on a single target class only, we aim to recover 3D mesh of broader classes with a single multi-task model. However, there exists no dataset that can directly enable multi-task learning due to the absence of both human and animal annotations for a single object, e.g., a human image does not have animal pose annotations; thus, we have to devise a new way to exploit heterogeneous datasets. To make the unstable disjoint multi-task learning jointly trainable, we propose to exploit the morphological similarity between humans and animals, motivated by animal exercise where humans imitate animal poses. We realize the morphological similarity by semantic correspondences, called sub-keypoint, which enables joint training of human and animal mesh regression branches. Besides, we propose class-sensitive regularization methods to avoid a mean-shape bias and to improve the distinctiveness across multi-classes. Our method performs favorably against recent uni-modal models on various human and animal datasets while being far more compact.