Versatile Multi-Modal Pre-Training for Human-Centric Perception

TL;DR

HCMoCo is a versatile multi-modal pre-training framework for human-centric perception that effectively leverages diverse human data modalities and priors, improving downstream task performance especially in data-scarce scenarios.

Contribution

The paper introduces HCMoCo, a novel contrastive learning framework that hierarchically learns modal-invariant representations using dense and sparse contrastive objectives for multi-modal human data.

Findings

Significant improvements in DensePose Estimation and Human Parsing with 7.16% and 12% gains.

Effective cross-modality supervision and missing-modality inference demonstrated.

Versatility across multiple downstream tasks validated.

Abstract

Human-centric perception plays a vital role in vision and graphics. But their data annotations are prohibitively expensive. Therefore, it is desirable to have a versatile pre-train model that serves as a foundation for data-efficient downstream tasks transfer. To this end, we propose the Human-Centric Multi-Modal Contrastive Learning framework HCMoCo that leverages the multi-modal nature of human data (e.g. RGB, depth, 2D keypoints) for effective representation learning. The objective comes with two main challenges: dense pre-train for multi-modality data, efficient usage of sparse human priors. To tackle the challenges, we design the novel Dense Intra-sample Contrastive Learning and Sparse Structure-aware Contrastive Learning targets by hierarchically learning a modal-invariant latent space featured with continuous and ordinal feature distribution and structure-aware semantic consistency. HCMoCo provides pre-train for different modalities by combining heterogeneous datasets, which allows efficient usage of existing task-specific human data. Extensive experiments on four downstream tasks of different modalities demonstrate the effectiveness of HCMoCo, especially under data-efficient settings (7.16% and 12% improvement on DensePose Estimation and Human Parsing). Moreover, we demonstrate the versatility of HCMoCo by exploring cross-modality supervision and missing-modality inference, validating its strong ability in cross-modal association and reasoning.