Understandable Controller Extraction from Video Observations of Swarms

TL;DR

This paper presents a method to automatically extract understandable swarm controllers from video observations using evolutionary algorithms, enabling analysis and replication of swarm behaviors from visual data.

Contribution

It introduces a novel approach to derive swarm control rules directly from videos, bridging the gap between observed behaviors and underlying local interaction rules.

Findings

Successfully extracted multiple controllers for a collective movement task.

Controllers produced behaviors with similar outcomes but different underlying rules.

First demonstration of automatic swarm controller extraction from visual observations.

Abstract

Swarm behavior emerges from the local interaction of agents and their environment often encoded as simple rules. Extracting the rules by watching a video of the overall swarm behavior could help us study and control swarm behavior in nature, or artificial swarms that have been designed by external actors. It could also serve as a new source of inspiration for swarm robotics. Yet extracting such rules is challenging as there is often no visible link between the emergent properties of the swarm and their local interactions. To this end, we develop a method to automatically extract understandable swarm controllers from video demonstrations. The method uses evolutionary algorithms driven by a fitness function that compares eight high-level swarm metrics. The method is able to extract many controllers (behavior trees) in a simple collective movement task. We then provide a qualitative analysis of behaviors that resulted in different trees, but similar behaviors. This provides the first steps toward automatic extraction of swarm controllers based on observations.