XPath Agent: An Efficient XPath Programming Agent Based on LLM for Web Crawler

TL;DR

XPath Agent is a novel web crawling tool that automatically generates XPath queries from natural language, reducing manual effort and improving efficiency in web GUI testing and crawling tasks.

Contribution

It introduces a two-stage pipeline for automatic XPath query generation from natural language, enhancing efficiency and integration in web crawling workflows.

Findings

Achieves comparable performance to state-of-the-art methods

Reduces token usage significantly

Improves clock-time efficiency

Abstract

We present XPath Agent, a production-ready XPath programming agent specifically designed for web crawling and web GUI testing. A key feature of XPath Agent is its ability to automatically generate XPath queries from a set of sampled web pages using a single natural language query. To demonstrate its effectiveness, we benchmark XPath Agent against a state-of-the-art XPath programming agent across a range of web crawling tasks. Our results show that XPath Agent achieves comparable performance metrics while significantly reducing token usage and improving clock-time efficiency. The well-designed two-stage pipeline allows for seamless integration into existing web crawling or web GUI testing workflows, thereby saving time and effort in manual XPath query development. The source code for XPath Agent is available at https://github.com/eavae/feilian.