DatasetAgent: A Novel Multi-Agent System for Auto-Constructing Datasets from Real-World Images

TL;DR

DatasetAgent is a multi-agent system that automates the creation of high-quality image datasets from real-world images, reducing manual effort and improving dataset quality for training vision models.

Contribution

It introduces a novel multi-agent framework utilizing multimodal large language models to automatically construct and optimize image datasets from real-world images.

Findings

Effective in expanding existing datasets and creating new ones from scratch.

Datasets generated by DatasetAgent improve training of vision models for classification, detection, and segmentation.

Demonstrates high-quality dataset construction from diverse open-source data.

Abstract

Common knowledge indicates that the process of constructing image datasets usually depends on the time-intensive and inefficient method of manual collection and annotation. Large models offer a solution via data generation. Nonetheless, real-world data are obviously more valuable comparing to artificially intelligence generated data, particularly in constructing image datasets. For this reason, we propose a novel method for auto-constructing datasets from real-world images by a multiagent collaborative system, named as DatasetAgent. By coordinating four different agents equipped with Multi-modal Large Language Models (MLLMs), as well as a tool package for image optimization, DatasetAgent is able to construct high-quality image datasets according to user-specified requirements. In particular, two types of experiments are conducted, including expanding existing datasets and creating new ones from scratch, on a variety of open-source datasets. In both cases, multiple image datasets constructed by DatasetAgent are used to train various vision models for image classification, object detection, and image segmentation.