You Only Submit One Image to Find the Most Suitable Generative Model

TL;DR

This paper introduces a novel method for efficiently identifying the most suitable generative model for a user's needs using only one example image, significantly improving model selection accuracy.

Contribution

It proposes a comprehensive solution with three modules for generative model identification, addressing the challenge of matching user requirements with large model repositories.

Findings

Achieves over 80% top-4 identification accuracy with a single image input

Demonstrates efficiency and effectiveness of the proposed approach

Addresses cross-modality and dimensionality challenges in model identification

Abstract

Deep generative models have achieved promising results in image generation, and various generative model hubs, e.g., Hugging Face and Civitai, have been developed that enable model developers to upload models and users to download models. However, these model hubs lack advanced model management and identification mechanisms, resulting in users only searching for models through text matching, download sorting, etc., making it difficult to efficiently find the model that best meets user requirements. In this paper, we propose a novel setting called Generative Model Identification (GMI), which aims to enable the user to identify the most appropriate generative model(s) for the user's requirements from a large number of candidate models efficiently. To our best knowledge, it has not been studied yet. In this paper, we introduce a comprehensive solution consisting of three pivotal modules: a weighted Reduced Kernel Mean Embedding (RKME) framework for capturing the generated image distribution and the relationship between images and prompts, a pre-trained vision-language model aimed at addressing dimensionality challenges, and an image interrogator designed to tackle cross-modality issues. Extensive empirical results demonstrate the proposal is both efficient and effective. For example, users only need to submit a single example image to describe their requirements, and the model platform can achieve an average top-4 identification accuracy of more than 80%.