A Comprehensive Dataset for Human vs. AI Generated Text Detection

TL;DR

This paper introduces a large, diverse dataset of over 58,000 texts combining authentic news articles and AI-generated versions from multiple models, aiming to improve detection and attribution of AI-generated text.

Contribution

It provides a comprehensive, well-annotated dataset bridging real-world journalism and AI-generated content to facilitate detection and attribution research.

Findings

Baseline detection accuracy of 58.35% for distinguishing human vs. AI text.

Attribution accuracy of 8.92% for identifying the specific AI model.

Dataset includes diverse models and real news content for robust evaluation.

Abstract

The rapid advancement of large language models (LLMs) has led to increasingly human-like AI-generated text, raising concerns about content authenticity, misinformation, and trustworthiness. Addressing the challenge of reliably detecting AI-generated text and attributing it to specific models requires large-scale, diverse, and well-annotated datasets. In this work, we present a comprehensive dataset comprising over 58,000 text samples that combine authentic New York Times articles with synthetic versions generated by multiple state-of-the-art LLMs including Gemma-2-9b, Mistral-7B, Qwen-2-72B, LLaMA-8B, Yi-Large, and GPT-4-o. The dataset provides original article abstracts as prompts, full human-authored narratives. We establish baseline results for two key tasks: distinguishing human-written from AI-generated text, achieving an accuracy of 58.35\%, and attributing AI texts to their generating models with an accuracy of 8.92\%. By bridging real-world journalistic content with modern generative models, the dataset aims to catalyze the development of robust detection and attribution methods, fostering trust and transparency in the era of generative AI. Our dataset is available at: https://huggingface.co/datasets/gsingh1-py/train.