Benchmarking of LLM Detection: Comparing Two Competing Approaches

TL;DR

This paper evaluates various detectors for identifying LLM-generated text, highlighting the importance of standardized benchmarking datasets to fairly compare detection performance.

Contribution

It introduces a new evaluation dataset for benchmarking LLM detectors and compares the performance of existing detection approaches using this dataset.

Findings

Detection performance varies significantly across different detectors.

Benchmarking results are heavily influenced by the choice of dataset.

Standardized evaluation datasets are crucial for fair comparison.

Abstract

This article gives an overview of the field of LLM text recognition. Different approaches and implemented detectors for the recognition of LLM-generated text are presented. In addition to discussing the implementations, the article focuses on benchmarking the detectors. Although there are numerous software products for the recognition of LLM-generated text, with a focus on ChatGPT-like LLMs, the quality of the recognition (recognition rate) is not clear. Furthermore, while it can be seen that scientific contributions presenting their novel approaches strive for some kind of comparison with other approaches, the construction and independence of the evaluation dataset is often not comprehensible. As a result, discrepancies in the performance evaluation of LLM detectors are often visible due to the different benchmarking datasets. This article describes the creation of an evaluation dataset and uses this dataset to investigate the different detectors. The selected detectors are benchmarked against each other.