Applying Ensemble Methods to Model-Agnostic Machine-Generated Text Detection

TL;DR

This paper explores ensemble methods to improve the detection of machine-generated text, especially when the underlying language model is unknown, achieving high accuracy with simple statistics and supervised learning.

Contribution

It introduces ensemble techniques applied to DetectGPT outputs, enhancing model-agnostic detection accuracy without requiring prior knowledge of the generative model.

Findings

Summary statistics improve AUROC from 0.61 to 0.73

Supervised learning boosts AUROC to 0.94

Method maintains zero-shot capability with simple features

Abstract

In this paper, we study the problem of detecting machine-generated text when the large language model (LLM) it is possibly derived from is unknown. We do so by apply ensembling methods to the outputs from DetectGPT classifiers (Mitchell et al. 2023), a zero-shot model for machine-generated text detection which is highly accurate when the generative (or base) language model is the same as the discriminative (or scoring) language model. We find that simple summary statistics of DetectGPT sub-model outputs yield an AUROC of 0.73 (relative to 0.61) while retaining its zero-shot nature, and that supervised learning methods sharply boost the accuracy to an AUROC of 0.94 but require a training dataset. This suggests the possibility of further generalisation to create a highly-accurate, model-agnostic machine-generated text detector.