Towards an Understanding and Explanation for Mixed-Initiative Artificial Scientific Text Detection

TL;DR

This paper investigates the differences between machine-generated and human-written scientific texts, proposing a mixed-initiative detection approach that combines human expertise with machine analysis, supported by visual analytics and validated through case studies and user studies.

Contribution

It introduces a novel mixed-initiative workflow and visual analytics tool for scientific text detection, addressing interpretability and generalization challenges in AI-based detection methods.

Findings

Quantitative analysis of text differences

Effective human-AI collaborative detection workflow

Validated approach through case studies and user study

Abstract

Large language models (LLMs) have gained popularity in various fields for their exceptional capability of generating human-like text. Their potential misuse has raised social concerns about plagiarism in academic contexts. However, effective artificial scientific text detection is a non-trivial task due to several challenges, including 1) the lack of a clear understanding of the differences between machine-generated and human-written scientific text, 2) the poor generalization performance of existing methods caused by out-of-distribution issues, and 3) the limited support for human-machine collaboration with sufficient interpretability during the detection process. In this paper, we first identify the critical distinctions between machine-generated and human-written scientific text through a quantitative experiment. Then, we propose a mixed-initiative workflow that combines human experts' prior knowledge with machine intelligence, along with a visual analytics prototype to facilitate efficient and trustworthy scientific text detection. Finally, we demonstrate the effectiveness of our approach through two case studies and a controlled user study with proficient researchers. We also provide design implications for interactive artificial text detection tools in high-stakes decision-making scenarios.