The Availability and Persistence of Web References in D-Lib Magazine

TL;DR

This study investigates the long-term availability and stability of URLs cited in D-Lib Magazine, revealing that about 28% were initially inaccessible, with a half-life of approximately 10 years, and identifies factors influencing URL decay.

Contribution

It provides empirical data on URL persistence over time in digital library citations and analyzes factors affecting URL availability and content stability.

Findings

28% of URLs failed initially; 30% failed after 25 weeks.

Content remained relatively stable, with only 16% changing more than 1 KB.

Estimated URL half-life is approximately 10 years.

Abstract

We explore the availability and persistence of URLs cited in articles published in D-Lib Magazine. We extracted 4387 unique URLs referenced in 453 articles published from July 1995 to August 2004. The availability was checked three times a week for 25 weeks from September 2004 to February 2005. We found that approximately 28% of those URLs failed to resolve initially, and 30% failed to resolve at the last check. A majority of the unresolved URLs were due to 404 (page not found) and 500 (internal server error) errors. The content pointed to by the URLs was relatively stable; only 16% of the content registered more than a 1 KB change during the testing period. We explore possible factors which may cause a URL to fail by examining its age, path depth, top-level domain and file extension. Based on the data collected, we found the half-life of a URL referenced in a D-Lib Magazine article is approximately 10 years. We also found that URLs were more likely to be unavailable if they pointed to resources in the .net, .edu or country-specific top-level domain, used non-standard ports (i.e., not port 80), or pointed to resources with uncommon or deprecated extensions (e.g., .shtml, .ps, .txt).