Adapting the Hypercube Model to Archive Deferred Representations and Their Descendants

TL;DR

This paper presents a method to improve web archiving by discovering and crawling deferred representations and their descendants, addressing challenges posed by client-side technologies like JavaScript.

Contribution

It adapts the Hypercube model to effectively identify and archive client-side generated web page states and their embedded resources.

Findings

Average of 38.5 descendants per seed URI

70.9% of descendants reached via onclick events

Added 15.6 times more embedded resources than Heritrix

Abstract

The web is today's primary publication medium, making web archiving an important activity for historical and analytical purposes. Web pages are increasingly interactive, resulting in pages that are increasingly difficult to archive. Client-side technologies (e.g., JavaScript) enable interactions that can potentially change the client-side state of a representation. We refer to representations that load embedded resources via JavaScript as deferred representations. It is difficult to archive all of the resources in deferred representations and the result is archives with web pages that are either incomplete or that erroneously load embedded resources from the live web. We propose a method of discovering and crawling deferred representations and their descendants (representation states that are only reachable through client-side events). We adapt the Dincturk et al. Hypercube model to construct a model for archiving descendants, and we measure the number of descendants and requisite embedded resources discovered in a proof-of-concept crawl. Our approach identified an average of 38.5 descendants per seed URI crawled, 70.9% of which are reached through an onclick event. This approach also added 15.6 times more embedded resources than Heritrix to the crawl frontier, but at a rate that was 38.9 times slower than simply using Heritrix. We show that our dataset has two levels of descendants. We conclude with proposed crawl policies and an analysis of the storage requirements for archiving descendants.