Who Writes the Review, Human or AI?

TL;DR

This paper presents a transfer learning-based method to accurately distinguish between AI-generated and human-written book reviews, achieving high detection accuracy and exploring LLM capabilities.

Contribution

It introduces a transfer learning approach for cross-topic detection of AI-generated text, validated on a new dataset with high accuracy.

Findings

Detection accuracy of 96.86% for AI vs. human reviews

Effective transfer learning improves detection across topics

Demonstrates capabilities and limitations of large language models

Abstract

With the increasing use of Artificial Intelligence in Natural Language Processing, concerns have been raised regarding the detection of AI-generated text in various domains. This study aims to investigate this issue by proposing a methodology to accurately distinguish AI-generated and human-written book reviews. Our approach utilizes transfer learning, enabling the model to identify generated text across different topics while improving its ability to detect variations in writing style and vocabulary. To evaluate the effectiveness of the proposed methodology, we developed a dataset consisting of real book reviews and AI-generated reviews using the recently proposed Vicuna open-source language model. The experimental results demonstrate that it is feasible to detect the original source of text, achieving an accuracy rate of 96.86%. Our efforts are oriented toward the exploration of the capabilities and limitations of Large Language Models in the context of text identification. Expanding our knowledge in these aspects will be valuable for effectively navigating similar models in the future and ensuring the integrity and authenticity of human-generated content.