Modeling Updates of Scholarly Webpages Using Archived Data
Yasith Jayawardana, Alexander C. Nwala, Gavindya Jayawardena, Jian Wu,, Sampath Jayarathna, Michael L. Nelson, C. Lee Giles

TL;DR
This paper presents a method to model webpage update dynamics using archived data, improving crawl strategies for scholarly web pages by estimating update frequencies with limited resources.
Contribution
It introduces a novel approach to estimate webpage update frequencies from archived data, enhancing crawl efficiency for scholarly web content.
Findings
Archived data can accurately estimate short-term update frequencies.
The proposed method outperforms baseline models in resource efficiency.
Optimized crawling strategies improve coverage and freshness of scholarly web pages.
Abstract
The vastness of the web imposes a prohibitive cost on building large-scale search engines with limited resources. Crawl frontiers thus need to be optimized to improve the coverage and freshness of crawled content. In this paper, we propose an approach for modeling the dynamics of change in the web using archived copies of webpages. To evaluate its utility, we conduct a preliminary study on the scholarly web using 19,977 seed URLs of authors' homepages obtained from their Google Scholar profiles. We first obtain archived copies of these webpages from the Internet Archive (IA), and estimate when their actual updates occurred. Next, we apply maximum likelihood to estimate their mean update frequency () values. Our evaluation shows that values derived from a short history of archived data provide a good estimate for the true update frequency in the short-term, and that…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
