Preprint: Poster: Did I Just Browse A Website Written by LLMs?

TL;DR

This paper introduces a scalable, highly accurate method for detecting websites generated by large language models, revealing their increasing prevalence and potential impact on the web ecosystem.

Contribution

It proposes a novel website-level classification pipeline that outperforms existing detectors on complex web content and provides the first large-scale analysis of LLM-dominant sites.

Findings

Achieved 100% accuracy on ground truth datasets.

Detected a significant portion of LLM-dominant sites in web archives.

Found increasing prevalence of LLM-generated content in search results.

Abstract

Increasingly, web content is automatically generated by large language models (LLMs) with little human input. We call this "LLM-dominant" content. Since LLMs plagiarize and hallucinate, LLM-dominant content can be unreliable and unethical. Yet, websites rarely disclose such content, and human readers struggle to distinguish it. Thus, we must develop reliable detectors for LLM-dominant content. However, state-of-the-art LLM detectors are inaccurate on web content, because web content has low positive rates, complex markup, and diverse genres, instead of clean, prose-like benchmark data SoTA detectors are optimized for. We propose a highly reliable, scalable pipeline that classifies entire websites. Instead of naively classifying text extracted from each page, we classify each site based on an LLM text detector's outputs of multiple prose-like pages to boost accuracies. We train and evaluate our detector by collecting 2 distinct ground truth datasets totaling 120 sites, and obtain 100% accuracies testing across them. In the wild, we detect a sizable portion of sites as LLM-dominant among 10k sites in search engine results and 10k in Common Crawl archives. We find LLM-dominant sites are growing in prevalence and rank highly in search results, raising questions about their impact on end users and the overall Web ecosystem.