The Hourglass Effect in Hierarchical Dependency Networks

TL;DR

This paper investigates the hourglass effect in hierarchical dependency networks, identifying the core modules that connect many inputs and outputs, and introduces a Reuse Preference model to explain this phenomenon across various systems.

Contribution

It generalizes the hourglass effect beyond layered networks, proposes a new model for module reuse bias, and empirically demonstrates the effect in diverse real-world networks.

Findings

All studied networks exhibit the hourglass property.

Core size varies across different networks.

The Reuse Preference model explains the hourglass effect.

Abstract

Many hierarchically modular systems are structured in a way that resembles an hourglass. This "hourglass effect" means that the system generates many outputs from many inputs through a relatively small number of intermediate modules that are critical for the operation of the entire system, referred to as the waist of the hourglass. We investigate the hourglass effect in general, not necessarily layered, hierarchical dependency networks. Our analysis focuses on the number of source-to-target dependency paths that traverse each vertex, and it identifies the core of a dependency network as the smallest set of vertices that collectively cover almost all dependency paths. We then examine if a given network exhibits the hourglass property or not, comparing its core size with a "flat" (i.e., non-hierarchical) network that preserves the source dependencies of each target in the original network. As a possible explanation for the hourglass effect, we propose the Reuse Preference (RP) model that captures the bias of new modules to reuse intermediate modules of similar complexity instead of connecting directly to sources or low complexity modules. We have applied the proposed framework in a diverse set of dependency networks from technological, natural and information systems, showing that all these networks exhibit the general hourglass property but to a varying degree and with different waist characteristics.