Intrinsically motivated collective motion

TL;DR

This paper demonstrates that collective motion behaviors like cohesion and alignment can naturally emerge from a low-level control principle called Future State Maximisation, which is inspired by potential evolutionary advantages in uncertain environments.

Contribution

It introduces FSM as a fundamental principle that explains collective motion emergence without explicitly encoding such behaviors, supported by neural network modeling and qualitative animal system similarities.

Findings

Collective behaviors emerge under FSM without explicit encoding.

Neural networks trained on trajectories mimic FSM heuristics.

FSM-based models resemble real animal group dynamics.

Abstract

Collective motion is found in various animal systems, active suspensions and robotic or virtual agents. This is often understood using high level models that directly encode selected empirical features, such as co-alignment and cohesion. Can these features be shown to emerge from an underlying, low-level principle? We find that they emerge naturally under Future State Maximisation (FSM). Here agents perceive a visual representation of the world around them, such as might be recorded on a simple retina, and then move to maximise the number of different visual environments that they expect to be able to access in the future. Such a control principle may confer evolutionary fitness in an uncertain world by enabling agents to deal with a wide variety of future scenarios. The collective dynamics that spontaneously emerge under FSM resemble animal systems in several qualitative aspects, including cohesion, co-alignment and collision suppression, none of which are explicitly encoded in the model. A multi-layered neural network trained on simulated trajectories is shown to represent a heuristic mimicking FSM. Similar levels of reasoning would seem to be accessible under animal cognition, demonstrating a possible route to the emergence of collective motion in social animals directly from the control principle underlying FSM. Such models may also be good candidates for encoding into possible future realisations of artificial "intelligent" matter, able to sense light, process information and move.

Full text

\dates

This manuscript was compiled on

\dropcap

There have been notable recent advances in our understanding of collective motion motivated by thermodynamics or physical arguments (1, 2, 3, 4, 5, 6, 7, 8) and in animal systems (9, 10, 11, 12, 13). While generalised hydrodynamic theories (6, 7, 8) can be obtained for certain active physical systems, the collective motion of agents capable of information processing can be far more complex. For example, existing generalised hydrodynamic theories do not account for long-ranged interactions, such as are likely to arise in higher animals that rely on vision. Agent-based models have been developed that incorporate some of these potential complexities, e.g. distance-dependent attraction, orientation or repulsions (5, 14, 15) or those relating more directly to vision (16, 17). While these models have had some success in explaining animal data the starting point is usually an essentially empirical model. This leads to challenges, both in controlling against overfitting and providing low-level explanatory power, “Why and how do agents co-align or remain in cohesive groups?". This question is difficult to answer if the model has co-alignment and cohesion hard-wired into it for essentially empirical reasons.

We instead analyse an agent based system in which each agent senses, and then processes, information in the context of a predictive model of the future. It uses this model to determine its action in the present, re-computing its model of the future from scratch at each discrete time step. Each agent decides how to move according to a low-level motivational principle that we call Future State Maximisation (FSM): it seeks control in the sense that it maximises the variety of (visual) environments that an agent could access before some time horizon, $\tau$ time steps into the future. This is a form of control as it gives the agent many future options in a potentially uncertain world.

As we report below, FSM spontaneously generates collective motion of a sort that is similar to that observed in animal systems, i.e. moving, cohesive, highly aligned, swarms that are stable against small perturbations, see SI Movie 1. While there are even quantitative similarities with the structure and order in flocks of birds (9) the motivation for our work is not to mimic a particular animal system but rather to analyse a simple, low-level model that may provide a general conceptual basis for collective motion, here based on vision. Crucially, our model does not explicitly include co-alignment, cohesion or any other physical interaction, merely mutual visual perception between agents in infinite (2D) space.

There are several reasons why motivational principles like FSM, that loosely serve to keep options open, may confer fitness, either in artificial intelligence or in nature. FSM increases the control that an agent has over their future. Agents that have many options to re-position themselves relative to their neighbours, e.g. in response to the arrival of a predator, can likely better avoid or confuse that predator.

In general, strategies like FSM that preserve an agent’s freedom to reach many different outcomes in an uncertain world are expected to enhance fitness.

Similar strategies are known to be successful in games like chess. Having access to many viable future lines of development is generically preferable, given uncertainty about how the game will actually develop. This confers robustness in defence and strategic manoeuvrability in attack. Chess players are familiar with the feeling of their options becoming progressively more limited as they lose a game, with the converse being strongly characteristic of winning. One attempt at formalising this kind of principle is the “empowerment" framework which does so using the language of information theory (18, 19, 20). Our implementation probably has most in common with this strand of the literature. FSM is an example of an intrinsic motivation (21, 22) where an incentive for behaviour is provided even in the absence of any specific tasks to complete or immediate external rewards to be gained. Intrinsically motivated behaviour has been studied extensively in the psychology literature (23, 24) as well as more recently in the field of reinforcement learning (25, 26, 27) where it is used to aid exploration in environments where rewards are sparse. The key principle is that such behaviour should offer a generic and universal benefit to the agent, not because it is useful for solving any one particular problem, but because it is beneficial for a wide range of scenarios that the agent may encounter in the future. A similar idea arises in the analysis of (hypothetical) causal entropic forces (28, 29). These forces generate motion that increases an entropy-like measure of all paths into the future and can lead to behaviour with features usually thought to be characteristic of intelligence, including evidence for the spontaneous emergence of tool-use and social cooperation.

Other work on decision making has some similarities with FSM (30). In that study a formalism similar to (28) was used to model agents making a group-level decision: agents reach a consensus on a (single) decision; made in the same sensory context for all agents; without those agents perceiving states (more than a single step) in the future. No explicit dynamical model was defined or analysed in (30). In the present work, FSM is applied to a group of agents that can move, perceive their own distinct environments and build independent models of future states that are accessible to them, guiding their decision making. This leads to the emergence of rich collective dynamics of a kind not previously realised.

Our work can also be seen as motivating the development of artificial particles that can sense, compute and move; so-called “intelligent matter". This is a natural direction in which to develop existing active systems, e.g. phoretic colloids (31), swimming cells (32) or active biological suspensions (33) that have limited, and rigid, information processing capabilities. Having candidate algorithms to encode into this intelligent matter will help motivate its development. Heuristics that mimic FSM, as discussed below, may represent a particularly powerful choice for such algorithms.

FSM applied to collective motion

Methods

We use deterministic computer simulation to study the motion of agents executing FSM. These agents are unit radius, phantom (i.e. able to overlap without repulsion) circular disks that are free to move on an infinite 2D plane. Their speed is the distance moved in each unit time step, with all lengths measured in disk radius units. Fig 1(a) shows the movement options available to each agent at each time step. These options are taken relative to its direction of motion in the previous time step. They are, in order: continue in the same direction with a choice of three different speeds, $v_{0}$ (nominal), $v_{0}-\Delta v$ (slow) or $v_{0}+\Delta v$ (fast). Alternatively, they are able to turn left or right by a small angle $\Delta\theta$, with speed $v_{0}$. Unless specified otherwise in what follows the nominal speed $v_{0}=10$, the speed variation $\Delta v=2$ and the angular rotation $\Delta\theta=\text{$$}$. At each time step the agent must choose one of these five actions $z\in\{z_{1},z_{2},z_{3},z_{4},z_{5}\}$ and does so by executing a form of FSM, as described below.

Fig 1(b) shows how each visual state is constructed using the positions of the agents.

This visual state is constructed for each agent by geometrically projecting all $N-1$ other disks down onto its centre. This involves constructing pairs of lines that each pass through the focal disk’s centre and are tangent to both sides of the other disks. Each of these lines can be specified by an angle, measured relative to the agent’s direction of motion. This allows us to define angular regions in which a line of sight will intersect with one or more other disks, shown as solid blue regions on the perimeter of Fig 1(b). We construct $n_{s}$ discrete visual sensors that each relate to an angular region of size $2\pi/n_{s}$. The radial dotted lines in Fig 1(b) denote the angular sensors (not the tangent lines). Each sensor registers 1 if more than half occupied by angles along which a line of sight will intersect other disk(s), i.e. the fraction of solid blue; 0 otherwise. Unless stated otherwise $n_{s}=40$ in all simulations. Panel (c) shows how each agent constructs its future decision tree, given a model for the motion of all other agents, here simply that they continue on their previous trajectory at nominal speed $v_{o}$, as illustrated by the blue agent. In this way the agent can compare each of the five moves available to it based on the absolute number of different visual states on all nodes accessible from that move. It chooses the move that maximises this measure.

In more mathematical language, we define the visual state $\mathbf{f_{i}}\in\{0,1\}^{n_{s}}$ of an agent on the $i^{\rm th}$ node of its tree of potential future states, as discussed above. Each of the five available moves in the next time step leads to branch $\alpha$ of the tree of potential future states. For each of these five branches we then construct a set $S_{\alpha}$ consisting of all of the unique visual states $\mathbf{f_{i}^{(\alpha)}}$ within that branch. The future time horizon (tree depth) is $\tau=4$ in our simulations, unless stated otherwise. Each branch is then given a weight $W_{\alpha}=|S_{\alpha}|$, and the agent then chooses the current action $z_{\\ alpha^{*}}$ such that $\alpha^{*}=\operatorname*{\arg\!\max}_{\alpha}z_{\alpha}$.

Consider a toy example of this process in which there are only $n_{s}=4$ sensors and two possible actions. Imagine that the branch $\alpha=1$, following action $z_{1}$, leads to three nodes with visual states of $\{1,0,1,0\}$, $\{1,0,0,0\}$, $\{1,0,0,0\}$ and $\{1,0,1,0\}$, while the branch $\alpha=2$, following action $z_{2}$, leads to four nodes with visual states of $\{1,0,1,0\}$, $\{1,0,0,0\}$, $\{1,1,0,0\}$ and $\{1,0,1,1\}$. In this example, branch $\alpha=2$, and hence action $z_{2}$, would be chosen because it leads to a future with four distinct Boolean vectors (states) whereas the branch $\alpha=1$ contains only 2 distinct states; the vectors $\{1,0,1,0\}$ and $\{1,0,0,0\}$ being repeated.

Some nodes on the decision tree correspond to collisions and are highlighted on Fig 1(b) with a dotted red outline. An agent considers any branch of its decision tree to terminate on collision, i.e. this and any subsequent nodes are deemed inaccessible. In this way the agent tends to avoid collisions because they contribute no states to its future. We find a strong reduction of collisions in the FSM trajectories that result, typically 2-3 orders of magnitude below a control collision rate (see Fig 2 in the SI). This is in spite of the fact that there is no explicit suppression of collisions, e.g. via physical interactions.

In the SI we discuss how to generalise this to a continuous measure of the degeneracy of future visual states.

In (28) a Gibbs measure of the accessible state space, rather than a count the number of distinct states, is used to quantify the future freedom. Our work could be extended in this direction in the future.

Results

Swarms similar to that shown in Fig 2(a) arise from these FSM dynamics across a broad range of parameter values, see SI for a comparison. However, there are some restrictions, e.g. the number of sensors can neither be too large (so that all states become unique) or too small (sensory resolution is lost) and the time horizon must be sufficiently long. For time horizons that are too short ($\tau<4$ for $N=50$) the swarm becomes less stable with agents separating from the main swarm increasingly frequently. In general the initial conditions must be chosen to be roughly commensurate with the steady state. If the system is prepared in an initial configuration from which the agents’ decision trees cannot perceive the steady state within $\tau$ time steps then the swarm fragments, typically into cohesive subgroups, see SI Movie 2 for an example of this phenomenon with $N=500$. Such initial conditions correspond to widely separated and/or orientationally disordered agents. Robustness to variation of the initial conditions improves with increasing $\tau$.

The state shown in Fig 2(a) and in SI Movie 1 has further qualitative similarities with animal systems and, in particular, large flocks of starlings: Its alignment order parameter is within 1% of a typical value for starling flocks (9) and it is in a state of marginal opacity, in which the fraction of sensors in state 0 to state 1 is order unity (16) (see figure 2 in the SI for more details). Finally, the correlation length scales with the system size, as shown in panel (b). This is indicative of scale-free correlations, another feature of starling flocks (9), and systems close to criticality more generally(34). Fig 2(c) shows snapshots of a larger swarm ($N=500$, $\tau=5$), sequentially in time, with motion determined by FSM on the continuous measure of visual state degeneracy described in the SI. Whilst for smaller swarms the two approaches give virtually identical results (compare SI movies 1 and 3), for larger swarms the continuous measure has more variety in its steady state collective dynamics and is more robust to fragmentation (contrast SI movies 2 and 4).

It is perhaps somewhat counter-intuitive that such a highly ordered state emerges, given that FSM can be interpreted as preferring highly varied (roughly, high entropy) distributions of states. This is because FSM is insensitive to the variety/disorder of the swarm in the present. It is from such a highly ordered state that the swarm can access the greatest variety of states in the future. Thus it targets this state and remains there. The state is cohesive because nearby agents then have the most freedom to re-arrange their relative positions, Marginal opacity is selected because most configurations have sensor states roughly evenly split between 0 and 1.

Changing the heuristic used to model hypothetical future trajectories

A key ingredient of the FSM model is an assumption for how the other agents will move in the future. Without such an assumption their future positions remain undetermined and the corresponding visual projections cannot therefore be computed. Fig 1(c) shows the simplest of four different assumptions, or heuristics, that we report in this letter: all other agents (only a single (blue) one is shown) are assumed to continue on ballistic trajectories, without turning, at speed $v_{o}$. The structure of the cohesive, co-aligned swarms that spontaneously emerge under this assumption are shown in Fig 2(a) (see also SI Movie 1). The ballistic motion assumption is an approximate model for the motion of the other agents and is not strictly self-consistent insofar as all agents are identical and actually move according to FSM. Hence, the (other) agents won’t move in exactly such a ballistic fashion, as can be seen from the individual trajectories in Fig 2(a). Nonetheless, this assumption is quite good for the highly ordered (co-aligned) swarms that do emerge from FSM. All agents would indeed continue moving in exactly the same direction under perfect co-alignment. The alignment order parameter is here defined as $\phi=\langle\frac{1}{N}\sum_{i}^{N}\hat{\bf v}_{i}(t)\rangle$ with the average performed over time and $\hat{\bf v}_{i}(t)$ a unit vector in the direction of motion of the $i^{\rm th}$ agent at time step $t$. The swarm in Fig 2(a) has order $\phi\simeq 0.98$.

Other heuristics can be made self-consistent with FSM. Examples include: (A) Agents are assumed to collectively target a particular value of order. At each time step every agent, in random order, turns in either direction if this brings the collective order closer to the target order $\phi_{A}$, otherwise continuing at speed $v_{o}$. (B) Agents are assumed to move at speed $v_{o}$ according to a topological version of the Vicsek model (5), in which co-aligning neighbours are those that share edges under a Delaunay triangulation. As usual this model involves a variable noise $\eta$, with a one-to-one relationship between this and the average order parameter at that noise $\phi_{B}(\eta)$. Fig 4 shows that both these heuristics can be made self-consistent with FSM at the level of the order realized: The FSM trajectories that are generated, using these heuristics as a model for the motion of all (other) agents, then have the same order as is generated by the bare heuristic, a value that was not known a priori. Any evolutionary pressure to adopt FSM-like dynamics should, presumably, also favour the ability to self-consistently predict the behaviour of other members of the group in this way.

Training a neural network to mimic the FSM algorithm

While the full FSM algorithm is computationally demanding, an artificial neural network could serve as an example of a heuristic that can closely mimic FSM, and fitness benefits arising therefrom. Crucially, once trained, it is computationally simple and fast. Similar levels of reasoning could be expected to operate under animal cognition. We do not claim that an artificial neural network would be a direct model for (wet) neural networks, even though the former field was indeed motivated by the latter. We only argue that reasoning with this level of complexity could be encoded in an animal brain. This heuristic, like the others described above, could also be used as a model for the behaviour of other agents during FSM.

We have trained a multi-layered neural network on the simulated trajectories that arise under FSM over 200,000 time steps, as sketched in Fig 5.

We gather training data by running the full FSM algorithm using nominal parameters ($N=50$, $\tau=4$, $n_{s}=40$, $\Delta\theta=\text{$$}$, $v_{0}=10$, $\Delta v=2$). Future visual states are computed under the assumption that other agents will move ballistically in their future trajectories, i.e. at speed $v_{0}$ in their current direction of motion. We generate 800 separate simulations, each with agents initially placed randomly in a square region with dimensions that vary between 80 and 160 disk radii. Each agent’s initial orientation is chosen randomly from a Gaussian distribution with mean orientation along the nominal $x$-direction and a standard deviation of $2\Delta\theta$. We chose these different initial conditions to provide representative examples of trajectories that recover from perturbations. This allows the trained network to make decisions that mimic FSM in situations that vary from the steady state, improving its robustness. In each of the simulations we record the current speed and the current and previous visual state of every agent at every time step, along with the actual decision made by the FSM algorithm in that situation (represented as an integer between 1 and 5). Note that including memory, via the previous visual state, is found to be crucial in order to train a network which qualitatively reproduces the behaviour of the full FSM algorithm. The training process is a supervised learning problem in which we have 10 million labelled example decisions each corresponding to a vector input of dimension 81 (2 $n_{s}+1$, for the speed) with each output an integer between 1 and 5.

The neural network architecture we employ consists of a hidden layer of 200 fully connected neurons connected to the input with three further fully connected hidden layers of sizes 100, 50 and 25 respectively. The last of these is connected to a softmax classifier which outputs an integer between 1 and 5 corresponding to the selected action. All of the hidden layers use the RelU activaton function. We trained the neural network on all of the data for 500 epochs using the ADAM optimizer under Keras with an initial learning rate of 0.0001.

The output from our artificial neural network is seen to closely mimic the FSM trajectories, see SI Movie 6.

In summary, we propose a form of intrinsically motivated collective motion based on Future State Maximization (FSM). This involves a minimal representation of vision in which agents seek to increase their control of the visual world around them. Specifically, they target being able to reach the greatest variety of future environments. The potential fitness benefits of this lie in the fact that it gives the agent freedom to access different outcomes in an uncertain world. Cohesive, ordered swarms that resemble natural animal systems spontaneously emerge under FSM. This behaviour can be encoded in heuristics, mimicking full FSM. A neural network is an example of the kind of heuristic that could mimic FSM under animal cognition, providing a possible route for the evolutionary selection of this behaviour. Such heuristics could also lie within the processing power of future realisations of “intelligent" materials that may incorporate sensors, as well as the ability to move.

\acknow

This work was partially supported by the UK EPSRC though the Mathematics for Real-World Systems CDT, EP/L015374/1 (H.J.C.). Computing facilities were provided by the Scientific Computing Research Technology Platform of the University of Warwick. We acknowledge stimulating discussions with George Rowlands (Warwick) and Hugues Chaté (Saclay).

\showacknow