MultiHuman-Testbench: Benchmarking Image Generation for Multiple Humans

TL;DR

This paper introduces MultiHuman-Testbench, a comprehensive benchmark for evaluating generative models in creating multi-human images with complex actions, identities, and diversity, addressing a key gap in the field.

Contribution

It provides a new benchmark dataset, an evaluation suite with multiple metrics, and novel techniques for improving identity preservation in multi-human image generation.

Findings

Zero-shot and training-based models evaluated with diverse results.

Region isolation techniques significantly improve ID similarity.

Benchmark facilitates standardized assessment of multi-human image generation.

Abstract

Generation of images containing multiple humans, performing complex actions, while preserving their facial identities, is a significant challenge. A major factor contributing to this is the lack of a dedicated benchmark. To address this, we introduce MultiHuman-Testbench, a novel benchmark for rigorously evaluating generative models for multi-human generation. The benchmark comprises 1,800 samples, including carefully curated text prompts, describing a range of simple to complex human actions. These prompts are matched with a total of 5,550 unique human face images, sampled uniformly to ensure diversity across age, ethnic background, and gender. Alongside captions, we provide human-selected pose conditioning images which accurately match the prompt. We propose a multi-faceted evaluation suite employing four key metrics to quantify face count, ID similarity, prompt alignment, and action detection. We conduct a thorough evaluation of a diverse set of models, including zero-shot approaches and training-based methods, with and without regional priors. We also propose novel techniques to incorporate image and region isolation using human segmentation and Hungarian matching, significantly improving ID similarity. Our proposed benchmark and key findings provide valuable insights and a standardized tool for advancing research in multi-human image generation. The dataset and evaluation codes will be available at https://github.com/Qualcomm-AI-research/MultiHuman-Testbench.