Group-Adaptive Threshold Optimization for Robust AI-Generated Text Detection

TL;DR

This paper introduces FairOPT, a group-specific threshold optimization algorithm that improves the robustness and fairness of AI-generated text detectors across diverse text attributes, reducing misclassification disparities.

Contribution

The paper proposes a novel group-adaptive threshold optimization method, FairOPT, to address distributional biases in AI-text detection, enhancing fairness and robustness.

Findings

Significant reduction in detection discrepancy across subgroups.

Decreased overall discrepancy by 27.4% across multiple metrics.

Minimal impact on detection accuracy, only 0.005% sacrifice.

Abstract

The advancement of large language models (LLMs) has made it difficult to differentiate human-written text from AI-generated text. Several AI-text detectors have been developed in response, which typically utilize a fixed global threshold (e.g., $\theta = 0.5$) to classify machine-generated text. However, one universal threshold could fail to account for distributional variations by subgroups. For example, when using a fixed threshold, detectors make more false positive errors on shorter human-written text, and more positive classifications of neurotic writing styles among long texts. These discrepancies can lead to misclassifications that disproportionately affect certain groups. We address this critical limitation by introducing FairOPT, an algorithm for group-specific threshold optimization for probabilistic AI-text detectors. We partitioned data into subgroups based on attributes (e.g., text length and writing style) and implemented FairOPT to learn decision thresholds for each group to reduce discrepancy. FairOPT showed notable discrepancy mitigation across nine detectors and three heterogeneous datasets, and the remarkable mitigation of the minimax problem by decreasing overall discrepancy 27.4% across five metrics while minimally sacrificing accuracy by 0.005%. Our framework paves the way for more robust classification in AI-generated content detection via post-processing. We release our data, code, and project information at URL.