Adaptive Ensembles of Fine-Tuned Transformers for LLM-Generated Text Detection

TL;DR

This paper evaluates transformer-based models for detecting LLM-generated text, demonstrating that adaptive ensemble methods significantly enhance accuracy and generalization across diverse datasets.

Contribution

It introduces adaptive ensemble algorithms to improve the detection of LLM-generated text, outperforming single models especially on out-of-distribution data.

Findings

Single models perform well on in-distribution data but poorly on out-of-distribution data.

Adaptive ensemble methods significantly improve detection accuracy.

Ensemble models show strong generalization ability across datasets.

Abstract

Large language models (LLMs) have reached human-like proficiency in generating diverse textual content, underscoring the necessity for effective fake text detection to avoid potential risks such as fake news in social media. Previous research has mostly tested single models on in-distribution datasets, limiting our understanding of how these models perform on different types of data for LLM-generated text detection task. We researched this by testing five specialized transformer-based models on both in-distribution and out-of-distribution datasets to better assess their performance and generalizability. Our results revealed that single transformer-based classifiers achieved decent performance on in-distribution dataset but limited generalization ability on out-of-distribution dataset. To improve it, we combined the individual classifiers models using adaptive ensemble algorithms, which improved the average accuracy significantly from 91.8% to 99.2% on an in-distribution test set and from 62.9% to 72.5% on an out-of-distribution test set. The results indicate the effectiveness, good generalization ability, and great potential of adaptive ensemble algorithms in LLM-generated text detection.