Evolutionary Dynamics of the World Wide Web

TL;DR

This paper develops a stochastic growth model for the Web, predicting a universal power law distribution of site sizes, confirmed by large-scale data analysis, revealing scale invariance and enabling size estimations without exhaustive crawling.

Contribution

It introduces a new theoretical model for Web growth that accounts for stochastic site creation and growth rates, predicting a universal power law distribution.

Findings

Confirmed power law distribution of site sizes in Web data

Web exhibits scale invariance with no characteristic size

Model enables estimation of site counts without full Web crawl

Abstract

We present a theory for the growth dynamics of the World Wide Web that takes into account the wide range of stochastic growth rates in the number of pages per site, as well as the fact that new sites are created at different times. This leads to the prediction of a universal power law in the distribution of the number of pages per site which we confirm experimentally by analyzing data from large crawls made by the search engines Alexa and Infoseek. The existence of this power law not only implies the lack of any length scale for the Web, but also allows one to determine the expected number of sites of any given size without having to exhaustively crawl the Web.