Stylometry Analysis of Human and Machine Text for Academic Integrity

TL;DR

This paper presents a comprehensive NLP-based framework for distinguishing human from machine-generated text and detecting author changes to uphold academic integrity, evaluated on datasets with varying prompt strictness.

Contribution

It introduces a multi-task analysis of human and machine text, including author attribution and style change detection, with publicly available datasets and code for future research.

Findings

Performance drops on strict prompt datasets highlight detection challenges.

Proposed methods effectively differentiate human and machine text.

Datasets and tools are publicly accessible for benchmarking.

Abstract

This work addresses critical challenges to academic integrity, including plagiarism, fabrication, and verification of authorship of educational content, by proposing a Natural Language Processing (NLP)-based framework for authenticating students' content through author attribution and style change detection. Despite some initial efforts, several aspects of the topic are yet to be explored. In contrast to existing solutions, the paper provides a comprehensive analysis of the topic by targeting four relevant tasks, including (i) classification of human and machine text, (ii) differentiating in single and multi-authored documents, (iii) author change detection within multi-authored documents, and (iv) author recognition in collaboratively produced documents. The solutions proposed for the tasks are evaluated on two datasets generated with Gemini using two different prompts, including a normal and a strict set of instructions. During experiments, some reduction in the performance of the proposed solutions is observed on the dataset generated through the strict prompt, demonstrating the complexities involved in detecting machine-generated text with cleverly crafted prompts. The generated datasets, code, and other relevant materials are made publicly available on GitHub, which are expected to provide a baseline for future research in the domain.