DALD: Improving Logits-based Detector without Logits from Black-box LLMs

TL;DR

DALD introduces a novel framework that enhances black-box LLM detection by aligning surrogate model distributions with unknown target models, improving robustness without requiring logits from proprietary models.

Contribution

The paper proposes DALD, a distribution alignment method that improves logits-based detection of LLMs without access to logits, addressing distribution mismatch issues.

Findings

DALD outperforms existing methods in black-box LLM detection.

DALD maintains high detection accuracy across different model versions.

DALD requires minimal training data from publicly available outputs.

Abstract

The advent of Large Language Models (LLMs) has revolutionized text generation, producing outputs that closely mimic human writing. This blurring of lines between machine- and human-written text presents new challenges in distinguishing one from the other a task further complicated by the frequent updates and closed nature of leading proprietary LLMs. Traditional logits-based detection methods leverage surrogate models for identifying LLM-generated content when the exact logits are unavailable from black-box LLMs. However, these methods grapple with the misalignment between the distributions of the surrogate and the often undisclosed target models, leading to performance degradation, particularly with the introduction of new, closed-source models. Furthermore, while current methodologies are generally effective when the source model is identified, they falter in scenarios where the model version remains unknown, or the test set comprises outputs from various source models. To address these limitations, we present Distribution-Aligned LLMs Detection (DALD), an innovative framework that redefines the state-of-the-art performance in black-box text detection even without logits from source LLMs. DALD is designed to align the surrogate model's distribution with that of unknown target LLMs, ensuring enhanced detection capability and resilience against rapid model iterations with minimal training investment. By leveraging corpus samples from publicly accessible outputs of advanced models such as ChatGPT, GPT-4 and Claude-3, DALD fine-tunes surrogate models to synchronize with unknown source model distributions effectively.