Measuring Human Involvement in AI-Generated Text: A Case Study on Academic Writing

TL;DR

This paper introduces a novel method using BERTScore and a multi-task RoBERTa regressor to measure human involvement in AI-generated academic texts, addressing the limitations of binary detection methods.

Contribution

It proposes a new approach to quantify human involvement in AI-generated content, moving beyond binary detection to a continuous measurement framework.

Findings

The proposed method achieved an F1 score of 0.9423 in detecting human involvement.

It demonstrated strong generalizability across different generative models.

Existing detectors failed to accurately measure human involvement, highlighting the novelty of this approach.

Abstract

Content creation has dramatically progressed with the rapid advancement of large language models like ChatGPT and Claude. While this progress has greatly enhanced various aspects of life and work, it has also negatively affected certain areas of society. A recent survey revealed that nearly 30% of college students use generative AI to help write academic papers and reports. Most countermeasures treat the detection of AI-generated text as a binary classification task and thus lack robustness. This approach overlooks human involvement in the generation of content even though human-machine collaboration is becoming mainstream. Besides generating entire texts, people may use machines to complete or revise texts. Such human involvement varies case by case, which makes binary classification a less than satisfactory approach. We refer to this situation as participation detection obfuscation. We propose using BERTScore as a metric to measure human involvement in the generation process and a multi-task RoBERTa-based regressor trained on a token classification task to address this problem. To evaluate the effectiveness of this approach, we simulated academic-based scenarios and created a continuous dataset reflecting various levels of human involvement. All of the existing detectors we examined failed to detect the level of human involvement on this dataset. Our method, however, succeeded (F1 score of 0.9423 and a regressor mean squared error of 0.004). Moreover, it demonstrated some generalizability across generative models. Our code is available at https://github.com/gyc-nii/CAS-CS-and-dual-head-detector