Human-Guided Image Generation for Expanding Small-Scale Training Image Datasets

TL;DR

This paper introduces a human-guided image generation approach that enables controllable dataset expansion for small-scale datasets, improving model performance and user experience through sample-level prompt refinement.

Contribution

It proposes a multi-modal projection method with theoretical guarantees and a sample-level prompt refinement technique to facilitate controllable and user-friendly dataset augmentation.

Findings

Enhanced dataset diversity through human-guided generation

Improved classification and detection performance

Positive expert feedback on the method's usability

Abstract

The performance of computer vision models in certain real-world applications (e.g., rare wildlife observation) is limited by the small number of available images. Expanding datasets using pre-trained generative models is an effective way to address this limitation. However, since the automatic generation process is uncontrollable, the generated images are usually limited in diversity, and some of them are undesired. In this paper, we propose a human-guided image generation method for more controllable dataset expansion. We develop a multi-modal projection method with theoretical guarantees to facilitate the exploration of both the original and generated images. Based on the exploration, users refine the prompts and re-generate images for better performance. Since directly refining the prompts is challenging for novice users, we develop a sample-level prompt refinement method to make it easier. With this method, users only need to provide sample-level feedback (e.g., which samples are undesired) to obtain better prompts. The effectiveness of our method is demonstrated through the quantitative evaluation of the multi-modal projection method, improved model performance in the case study for both classification and object detection tasks, and positive feedback from the experts.