Webscraper: Leverage Multimodal Large Language Models for Index-Content Web Scraping

TL;DR

Webscraper introduces a novel framework using multimodal large language models to autonomously navigate and extract structured data from dynamic, interactive websites, surpassing traditional scraping methods.

Contribution

It presents a new multimodal LLM-based approach with a structured prompting procedure and custom tools for effective web content extraction from modern sites.

Findings

Achieves higher extraction accuracy than baseline methods.

Successfully applied to news and e-commerce websites.

Demonstrates robustness on dynamic, interactive web pages.

Abstract

Modern web scraping struggles with dynamic, interactive websites that require more than static HTML parsing. Current methods are often brittle and require manual customization for each site. To address this, we introduce Webscraper, a framework designed to handle the challenges of modern, dynamic web applications. It leverages a Multimodal Large Language Model (MLLM) to autonomously navigate interactive interfaces, invoke specialized tools, and perform structured data extraction in environments where traditional scrapers are ineffective. Webscraper utilizes a structured five-stage prompting procedure and a set of custom-built tools to navigate and extract data from websites following the common ``index-and-content'' architecture. Our experiments, conducted on six news websites, demonstrate that the full Webscraper framework, equipped with both our guiding prompt and specialized tools, achieves a significant improvement in extraction accuracy over the baseline agent Anthropic's Computer Use. We also applied the framework to e-commerce platforms to validate its generalizability.