NEXT-EVAL: Next Evaluation of Traditional and LLM Web Data Record Extraction

TL;DR

This paper introduces a comprehensive evaluation framework for web data record extraction, enabling fair comparison of traditional algorithms and LLM-based methods across diverse datasets with improved metrics and input formats.

Contribution

It presents a new evaluation framework with dataset generation, annotation, and structure-aware metrics, along with preprocessing strategies and a synthetic dataset for benchmarking extraction methods.

Findings

LLM with Flat JSON input achieves F1 score of 0.9567

Flat JSON input reduces hallucination in LLM extractions

Benchmarking shows LLMs outperform traditional algorithms with the new framework

Abstract

Effective evaluation of web data record extraction methods is crucial, yet hampered by static, domain-specific benchmarks and opaque scoring practices. This makes fair comparison between traditional algorithmic techniques, which rely on structural heuristics, and Large Language Model (LLM)-based approaches, offering zero-shot extraction across diverse layouts, particularly challenging. To overcome these limitations, we introduce a concrete evaluation framework. Our framework systematically generates evaluation datasets from arbitrary MHTML snapshots, annotates XPath-based supervision labels, and employs structure-aware metrics for consistent scoring, specifically preventing text hallucination and allowing only for the assessment of positional hallucination. It also incorporates preprocessing strategies to optimize input for LLMs while preserving DOM semantics: HTML slimming, Hierarchical JSON, and Flat JSON. Additionally, we created a publicly available synthetic dataset by transforming DOM structures and modifying content. We benchmark deterministic heuristic algorithms and off-the-shelf LLMs across these multiple input formats. Our benchmarking shows that Flat JSON input enables LLMs to achieve superior extraction accuracy (F1 score of 0.9567) and minimal hallucination compared to other input formats like Slimmed HTML and Hierarchical JSON. We establish a standardized foundation for rigorous benchmarking, paving the way for the next principled advancements in web data record extraction.