AI Generated Text Detection

TL;DR

This paper evaluates various AI-generated text detection methods using a new benchmark with diverse datasets, demonstrating that transformer-based models like DistilBERT outperform traditional approaches in accuracy and robustness.

Contribution

It introduces a unified benchmark for AI text detection with a topic-based data split to improve generalization, and compares traditional and transformer-based models on this benchmark.

Findings

Transformer models outperform traditional ML models.

DistilBERT achieves 88.11% accuracy and 0.96 ROC-AUC.

Topic-based data split enhances model robustness.

Abstract

The rapid development of large language models has led to an increase in AI-generated text, with students increasingly using LLM-generated content as their own work, which violates academic integrity. This paper presents an evaluation of AI text detection methods, including both traditional machine learning models and transformer-based architectures. We utilize two datasets, HC3 and DAIGT v2, to build a unified benchmark and apply a topic-based data split to prevent information leakage. This approach ensures robust generalization across unseen domains. Our experiments show that TF-IDF logistic regression achieves a reasonable baseline accuracy of 82.87%. However, deep learning models outperform it. The BiLSTM classifier achieves an accuracy of 88.86%, while DistilBERT achieves a similar accuracy of 88.11% with the highest ROC-AUC score of 0.96, demonstrating the strongest overall performance. The results indicate that contextual semantic modeling is significantly superior to lexical features and highlight the importance of mitigating topic memorization through appropriate evaluation protocols. The limitations of this work are primarily related to dataset diversity and computational constraints. In future work, we plan to expand dataset diversity and utilize parameter-efficient fine-tuning methods such as LoRA. We also plan to explore smaller or distilled models and employ more efficient batching strategies and hardware-aware optimization.