Beyond BeautifulSoup: Benchmarking LLM-Powered Web Scraping for Everyday Users

TL;DR

This paper benchmarks how large language models enable everyday users to perform web scraping tasks on complex sites, demonstrating that end-to-end LLM agents can automate data extraction with minimal prompts and effort.

Contribution

It systematically evaluates LLM-based web scraping workflows across diverse security measures, highlighting their accessibility and practical effectiveness for non-expert users.

Findings

End-to-end LLM agents can automate complex scraping with minimal prompts.

LLM-assisted scripting is effective for static sites and faster in some cases.

Users can achieve successful scraping with less than five prompt refinements.

Abstract

Web scraping has historically required technical expertise in HTML parsing, session management, and authentication circumvention, which limited large-scale data extraction to skilled developers. We argue that large language models (LLMs) have democratized web scraping, enabling low-skill users to execute sophisticated operations through simple natural language prompts. While extensive benchmarks evaluate these tools under optimal expert conditions, we show that without extensive manual effort, current LLM-based workflows allow novice users to scrape complex websites that would otherwise be inaccessible. We systematically benchmark what everyday users can do with off-the-shelf LLM tools across 35 sites spanning five security tiers, including authentication, anti-bot, and CAPTCHA controls. We devise and evaluate two distinct workflows: (a) LLM-assisted scripting, where users prompt LLMs to generate traditional scraping code but maintain manual execution control, and (b) end-to-end LLM agents, which autonomously navigate and extract data through integrated tool use. Our results demonstrate that end-to-end agents have made complex scraping accessible - requiring as little as a single prompt with minimal refinement (less than 5 changes) to complete workflows. We also highlight scenarios where LLM-assisted scripting may be simpler and faster for static sites. In light of these findings, we provide simple procedures for novices to use these workflows and gauge what adversaries could achieve using these.