Web graph compression with fast access

TL;DR

This paper compares existing web graph compression methods and introduces a new variant, aiming to improve memory efficiency and access speed for web graph representations.

Contribution

It evaluates state-of-the-art and existing algorithms, and proposes a new 2D stripes variant to enhance web graph compression and access performance.

Findings

The new 2D stripes method shows promising results.

Comparison reveals strengths and weaknesses of each technique.

The study provides insights for future web graph compression improvements.

Abstract

In recent years studying the content of the World Wide Web became a very important yet rather difficult task. There is a need for a compression technique that would allow a web graph representation to be put into the memory while maintaining random access time competitive to the time needed to access uncompressed web graph on a hard drive. There are already available techniques that accomplish this task, but there is still room for improvements and this thesis attempts to prove it. It includes a comparison of two methods contained in state of art of this field (BV and k2partitioned) to two already implemented algorithms (rewritten, however, in C++ programming language to maximize speed and resource management efficiency), which are LM and 2D, and introduces the new variant of the latter one, called 2D stripes. This thesis serves as well as a proof of concept. The final considerations show positive and negative aspects of all presented methods, expose the feasibility of the new variant as well as indicate future direction for development.