Dissecting Human Body Representations in Deep Networks Trained for Person Identification

TL;DR

This study analyzes how deep networks trained for person identification encode body and face features, revealing that face information influences accuracy and that embeddings contain gender, view, and dataset cues, with improvements possible via PCA-based subspace operations.

Contribution

It provides the first comprehensive analysis of body representations in long-term re-identification networks trained on large, unconstrained datasets, highlighting face influence and embedding manipulation techniques.

Findings

Face contributes to body identification accuracy.

Embeddings encode gender, view, and dataset information.

PCA-based subspace operations improve identification accuracy.

Abstract

Long-term body identification algorithms have emerged recently with the increased availability of high-quality training data. We seek to fill knowledge gaps about these models by analyzing body image embeddings from four body identification networks trained with 1.9 million images across 4,788 identities and 9 databases. By analyzing a diverse range of architectures (ViT, SWIN-ViT, CNN, and linguistically primed CNN), we first show that the face contributes to the accuracy of body identification algorithms and that these algorithms can identify faces to some extent -- with no explicit face training. Second, we show that representations (embeddings) generated by body identification algorithms encode information about gender, as well as image-based information including view (yaw) and even the dataset from which the image originated. Third, we demonstrate that identification accuracy can be improved without additional training by operating directly and selectively on the learned embedding space. Leveraging principal component analysis (PCA), identity comparisons were consistently more accurate in subspaces that eliminated dimensions that explained large amounts of variance. These three findings were surprisingly consistent across architectures and test datasets. This work represents the first analysis of body representations produced by long-term re-identification networks trained on challenging unconstrained datasets.