MUGC: Machine Generated versus User Generated Content Detection

TL;DR

This paper evaluates traditional machine learning methods for distinguishing machine-generated content from human-generated content across diverse datasets, highlighting the effectiveness of semantic and stylistic features in detection.

Contribution

It offers a comparative analysis of eight traditional algorithms and explores semantic and stylistic features for improved detection of machine-generated text.

Findings

High accuracy of traditional methods in identifying machine-generated data

Shorter length and less word variety in machine-generated texts

Deeper semantic features like word2vec improve detection performance

Abstract

As advanced modern systems like deep neural networks (DNNs) and generative AI continue to enhance their capabilities in producing convincing and realistic content, the need to distinguish between user-generated and machine generated content is becoming increasingly evident. In this research, we undertake a comparative evaluation of eight traditional machine-learning algorithms to distinguish between machine-generated and human-generated data across three diverse datasets: Poems, Abstracts, and Essays. Our results indicate that traditional methods demonstrate a high level of accuracy in identifying machine-generated data, reflecting the documented effectiveness of popular pre-trained models like RoBERT. We note that machine-generated texts tend to be shorter and exhibit less word variety compared to human-generated content. While specific domain-related keywords commonly utilized by humans, albeit disregarded by current LLMs (Large Language Models), may contribute to this high detection accuracy, we show that deeper word representations like word2vec can capture subtle semantic variances. Furthermore, readability, bias, moral, and affect comparisons reveal a discernible contrast between machine-generated and human generated content. There are variations in expression styles and potentially underlying biases in the data sources (human and machine-generated). This study provides valuable insights into the advancing capacities and challenges associated with machine-generated content across various domains.