Contrasting Web Robot and Human Behaviors with Network Models

TL;DR

This paper compares human and web robot browsing behaviors using session graph network models, revealing distinct traffic patterns that can improve web traffic modeling and caching strategies.

Contribution

It introduces the use of session graphs to differentiate human and robot web activity, providing insights for better traffic prediction and server optimization.

Findings

Human and robot session graphs have different degree distribution fits.

Distinct network properties reflect different traffic generation processes.

Results can inform improved web traffic models and caching algorithms.

Abstract

The web graph is a commonly-used network representation of the hyperlink structure of a website. A network of similar structure to the web graph, which we call the session graph has properties that reflect the browsing habits of the agents in the web server logs. In this paper, we apply session graphs to compare the activity of humans against web robots or crawlers. Understanding these properties will enable us to improve models of HTTP traffic, which can be used to predict and generate realistic traffic for testing and improving web server efficiency, as well as devising new caching algorithms. We apply large-scale network properties, such as the connectivity and degree distribution of human and Web robot session graphs in order to identify characteristics of the traffic which would be useful for modeling web traffic and improving cache performance. We find that the empirical degree distributions of session graphs for human and robot requests on one Web server are best fit by different theoretical distributions, indicating at a difference in the processes which generate the traffic.