Disrupting the Coming Robot Stampedes: Designing Resilient Information Ecologies

TL;DR

This paper discusses how dense, homogeneous machine networks can lead to dangerous mass behaviors and proposes ecologically-inspired diversity as a strategy to enhance resilience and prevent unintended large-scale failures.

Contribution

It introduces ecologically-based design principles, emphasizing diversity, to improve the resilience of interconnected machine systems against mass behavior risks.

Findings

Densely connected homogeneous systems risk collective failures.

Diversity in design can mitigate large-scale unintended behaviors.

Ecological principles offer effective strategies for resilient machine ecologies.

Abstract

Machines are designed to communicate widely and efficiently. Humans, less so. We evolved social structures that function best as small subgroups interacting within larger populations. Technology changes this dynamic, by allowing all individuals to be connected at the speed of light. A dense, tightly connected population can behave like a single agent. In animals, this happens in constrained areas where stampedes can easily form. Machines do not need these kinds of conditions. The very techniques used to design best-of-breed solutions may increase the risk of dangerous mass behaviors among homogeneous machines. In this paper we argue that ecologically-based design principles such as the presence of diversity are a broadly effective strategy to defend against unintended consequences at scale.