Is Synthetic Dataset Reliable for Benchmarking Generalizable Person Re-Identification?

TL;DR

This paper investigates whether synthetic datasets can reliably be used for benchmarking generalizable person re-identification algorithms, proposing a statistical analysis method to compare synthetic and real-world datasets.

Contribution

It introduces Pairwise Ranking Analysis (PRA), a statistical framework to evaluate the similarity of algorithm rankings across datasets, validating synthetic datasets for benchmarking.

Findings

Synthetic dataset ClonedPerson is statistically comparable to real-world datasets for benchmarking.

The proposed PRA method effectively measures ranking similarity between datasets.

Synthetic datasets can be reliably used for both training and testing in person ReID.

Abstract

Recent studies show that models trained on synthetic datasets are able to achieve better generalizable person re-identification (GPReID) performance than that trained on public real-world datasets. On the other hand, due to the limitations of real-world person ReID datasets, it would also be important and interesting to use large-scale synthetic datasets as test sets to benchmark person ReID algorithms. Yet this raises a critical question: is synthetic dataset reliable for benchmarking generalizable person re-identification? In the literature there is no evidence showing this. To address this, we design a method called Pairwise Ranking Analysis (PRA) to quantitatively measure the ranking similarity and perform the statistical test of identical distributions. Specifically, we employ Kendall rank correlation coefficients to evaluate pairwise similarity values between algorithm rankings on different datasets. Then, a non-parametric two-sample Kolmogorov-Smirnov (KS) test is performed for the judgement of whether algorithm ranking correlations between synthetic and real-world datasets and those only between real-world datasets lie in identical distributions. We conduct comprehensive experiments, with ten representative algorithms, three popular real-world person ReID datasets, and three recently released large-scale synthetic datasets. Through the designed pairwise ranking analysis and comprehensive evaluations, we conclude that a recent large-scale synthetic dataset ClonedPerson can be reliably used to benchmark GPReID, statistically the same as real-world datasets. Therefore, this study guarantees the usage of synthetic datasets for both source training set and target testing set, with completely no privacy concerns from real-world surveillance data. Besides, the study in this paper might also inspire future designs of synthetic datasets.