XTreePath: A generalization of XPath to handle real world structural variation

TL;DR

XTreePath enhances information extraction by capturing contextual DOM node information and using recursive tree matching, significantly improving robustness against HTML template changes compared to traditional XPath methods.

Contribution

It introduces XTreePath, a novel annotation and matching approach that handles real-world structural variations in web data extraction.

Findings

XTreePath outperforms XPath in extraction success rate.

The method is effective across diverse websites and domains.

It demonstrates robustness to HTML template changes.

Abstract

We discuss a key problem in information extraction which deals with wrapper failures due to changing content templates. A good proportion of wrapper failures are due to HTML templates changing to cause wrappers to become incompatible after element inclusion or removal in a DOM (Tree representation of HTML). We perform a large-scale empirical analyses of the causes of shift and mathematically quantify the levels of domain difficulty based on entropy. We propose the XTreePath annotation method to captures contextual node information from the training DOM. We then utilize this annotation in a supervised manner at test time with our proposed Recursive Tree Matching method which locates nodes most similar in context recursively using the tree edit distance. The search is based on a heuristic function that takes into account the similarity of a tree compared to the structure that was present in the training data. We evaluate XTreePath using 117,422 pages from 75 diverse websites in 8 vertical markets. Our XTreePath method consistently outperforms XPath and a current commercial system in terms of successful extractions in a blackbox test. We make our code and datasets publicly available online.