Web Execution Bundles: Reproducible, Accurate, and Archivable Web Measurements

TL;DR

WebREC is a new tool and format for web measurements that enhances reproducibility, accuracy, and versatility across diverse web research tasks, addressing limitations of prior custom and inconsistent methods.

Contribution

The paper introduces WebREC and .web, novel tools that enable accurate, reusable, and comprehensive web measurements and archiving, improving reproducibility and correctness in web research.

Findings

WebREC accurately replicates existing studies, matching baseline results.

70% of surveyed web research tasks can be performed with WebREC without modifications.

48% of tasks can utilize .web archives directly, enhancing reproducibility.

Abstract

Recently, reproducibility has become a cornerstone in the security and privacy research community, including artifact evaluations and even a new symposium topic. However, Web measurements lack tools that can be reused across many measurement tasks without modification, while being robust to circumvention, and accurate across the wide range of behaviors in the Web. As a result, most measurement studies use custom tools and varied archival formats, each of unknown correctness and significant limitations, systematically affecting the research's accuracy and reproducibility. To address these limitations, we present WebREC, a Web measurement tool that is, compared against the current state-of-the-art, accurate (i.e., correctly measures and attributes events not possible with existing tools), general (i.e., reusable without modification for a broad range of measurement tasks), and comprehensive (i.e., handling events from all relevant browser behaviors). We also present .web, an archival format for the accurate and reproducible measurement of a wide range of website behaviors. We empirically evaluate WebREC's accuracy by replicating well-known Web measurement studies and showing that WebREC's results more accurately match our baseline. We then assess if WebREC and .web succeed as general-purpose tools, which could be used to accomplish many Web measurement tasks without modification. We find that this is so: 70% of papers discussed in a 2024 web crawling SoK paper could be conducted using WebREC as is, and a larger number (48%) could be leveraged against .web archives without requiring any new crawling.