RU-AI: A Large Multimodal Dataset for Machine-Generated Content Detection

TL;DR

This paper introduces RU-AI, a large multimodal dataset for detecting machine-generated content across text, image, and voice, aiming to improve detection methods amidst the rise of generative AI models.

Contribution

The creation of RU-AI, a comprehensive multimodal dataset with over 1.4 million instances, including noise variants, to facilitate research in machine-generated content detection.

Findings

Current state-of-the-art models struggle with accuracy and robustness on RU-AI.

The dataset highlights the need for improved detection techniques for multimodal AI-generated content.

Extensive experiments demonstrate the challenges in existing detection methods.

Abstract

The recent generative AI models' capability of creating realistic and human-like content is significantly transforming the ways in which people communicate, create and work. The machine-generated content is a double-edged sword. On one hand, it can benefit the society when used appropriately. On the other hand, it may mislead people, posing threats to the society, especially when mixed together with natural content created by humans. Hence, there is an urgent need to develop effective methods to detect machine-generated content. However, the lack of aligned multimodal datasets inhibited the development of such methods, particularly in triple-modality settings (e.g., text, image, and voice). In this paper, we introduce RU-AI, a new large-scale multimodal dataset for robust and effective detection of machine-generated content in text, image and voice. Our dataset is constructed on the basis of three large publicly available datasets: Flickr8K, COCO and Places205, by adding their corresponding AI duplicates, resulting in a total of 1,475,370 instances. In addition, we created an additional noise variant of the dataset for testing the robustness of detection models. We conducted extensive experiments with the current SOTA detection methods on our dataset. The results reveal that existing models still struggle to achieve accurate and robust detection on our dataset. We hope that this new data set can promote research in the field of machine-generated content detection, fostering the responsible use of generative AI. The source code and datasets are available at https://github.com/ZhihaoZhang97/RU-AI.